Assorted Problems with Emacs 29

This hack patch will fix the texinfo problem. There is a better
fix in a branch, however brz is a pain to get going due to
being python 2.

--- info/vm.texinfo 2021-06-23 18:14:20.000000000 -0700
+++ /Users/markd/adm/build/emacs/vm/trunk/info/vm.texinfo 2020-07-05 13:57:37.000000000 -0700
@@ -7088,10 +7088,10 @@
 According to the project page, ``this site exists to continue VM development
 after version 7.19 as a community project.''

-@cindex Ulrich Müller
+@cindex Ulrich Muller
 @cindex Uday S Reddy
 Currently, VM is maintained by a ``VM Development Team,'' consisting of
-Robert Widhopf-Fenk, Ulrich Müller and Uday S Reddy. Other potential members
+Robert Widhopf-Fenk, Ulrich Muller and Uday S Reddy. Other potential members
 are warmly welcomed. Robert Fenk has been inactive since November, 2008
 but he continues to be an official member of the team. The new
 releases made by the team are numbered @code{8.1.0} and up.

This can fix Mr. Müller namne (I would suggest Mueller if omitting umlaut), but I doubt It can do something for my lisp problems.

Today, when I issued the vm command, I got:

Debugger entered--Lisp error: (void-variable vm-update-virtual-messages)
  byte-code("\305\306\307\310#\311\306\312\313#\210\314\10\303\11\n\"\315\212\316\317\13\211\34\320H\321HJ)\")$\207" [vm-update-virtual-messages &key message-changing m message function-put vm-virtual-message-p speed -1 put byte-optimizer byte-compile-inline-expand defun* "Update all the virtual messages of M to reflect th..." mapc #f(compiled-function (v-m) #<bytecode 0x1d6c62574191b1c9>) 4 1] 10)
  require(vm-message)
  vm-session-initialization()
  vm()
  posta()
  funcall-interactively(posta)
  command-execute(posta)

The .emacs file contains these instructions about vm:

(load-library "vm-version")
(load-library "vm-misc")
(load-library "vm-folder")
(load-library "vm-vars")
(load-library "vm-virtual")
;;(load-library "vm-message")
(load-library "vm")

If I try to run the commented statement I get

"Symbol's value as variable is void: vm-update-virtual-messages"

I have a branch here where I am fixing problems related to emacs 28:
  https://code.launchpad.net/~markd-kermodei/vm/vm-emacs-28

These mostly related to the emacs JIT, and should fix the
vm-update-virtual-messages

It also correctly fixes the umlaut ;-)

Gian Uberto Lauri <email address hidden> writes:
> This can fix Mr. Müller namne (I would suggest Mueller if omitting
> umlaut), but I doubt It can do something for my lisp problems.
>
> Today, when I issued the vm command, I got:
>
> Debugger entered--Lisp error: (void-variable vm-update-virtual-messages)
> byte-code("\305\306\307\310#\311\306\312\313#\210\314\10\303\11\n\"\315\212\316\317\13\211\34\320H\321HJ)\")$\207" [vm-update-virtual-messages &key message-changing m message function-put vm-virtual-message-p speed -1 put byte-optimizer byte-compile-inline-expand defun* "Update all the virtual messages of M to reflect th..." mapc #f(compiled-function (v-m) #<bytecode 0x1d6c62574191b1c9>) 4 1] 10)
> require(vm-message)
> vm-session-initialization()
> vm()
> posta()
> funcall-interactively(posta)
> command-execute(posta)
>
>
> The .emacs file contains these instructions about vm:
>
> (load-library "vm-version")
> (load-library "vm-misc")
> (load-library "vm-folder")
> (load-library "vm-vars")
> (load-library "vm-virtual")
> ;;(load-library "vm-message")
> (load-library "vm")
>
>
> If I try to run the commented statement I get
>
> "Symbol's value as variable is void: vm-update-virtual-messages"
>
> --
> You received this bug notification because you are subscribed to VM.
> Matching subscriptions: vm bugs
> https://bugs.launchpad.net/bugs/1979048
>
> Title:
> Assorted Problems with Emacs 29
>
> Status in VM:
> New
>
> Bug description:
> The system is Ubuntu focal (20.04), Emacs is GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.32, cairo version 1.16.0) of 2022-05-16
> mouse-minibuffer-check: Minibuffer window is not active
>
> I downloaded vm-8.2.0b.tgz, un-tarred the package, ran configure, ran
> make.
>
> In make output (attached), there are several messages like the one
> below (the last one)
>
> vm-w3.el:51:16: Warning: the function ‘vm-mime-cid-retrieve’ might not
> be defined at runtime.
>
> then make fails on the texinfo part
>
> makeinfo vm.texinfo -o vm.info
> utf8 "\xFC" does not map to Unicode at /usr/share/texinfo/Texinfo/ParserNonXS.pm line 1796, <FH> line 6169.
> Malformed UTF-8 character: \xfc\x6c\x6c\x65\x72\x0a (unexpected non-continuation byte 0x6c, immediately after start byte 0xfc; need 6 bytes, got 1) in pattern match (m//) at /usr/share/texinfo/Texinfo/ParserNonXS.pm line 3364.
> Malformed UTF-8 character (fatal) at /usr/share/texinfo/Texinfo/ParserNonXS.pm line 3364.
> make[1]: *** [<builtin>: vm.info] Error 25
> make[1]: Leaving directory '/usr/local/share/emacs/site-lisp/vm-8.2.0b/info'
> make: *** [Makefile:37: all] Error 1
>
> but the .elc files are all there, and I am using vm since at least
> 2006, I have my consolidated routine and configuration, that would not
> be a problem.
>
> The problem is elsewhere, and is a blocking one preventing me to do my
> usu...

Got it, configured, compiled. Now the summary remains clear, no presentation buffer is generate and possibly no mail is downloaded.

I get this error when I press g (get new mail)

Debugger entered--Lisp error: (wrong-type-argument char-or-string-p nil)
  vm-imap-retrieve-to-target(#<process IMAP over SSL<1>> #<buffer inbox> [[nil 25273 34619 92656 2 vm-imap-report-retrieval-status (#3) nil 645000 nil] nil "<email address hidden>:[sorry, not here]:inbox" 0 522 nil nil nil nil nil nil nil] (IMAP4REV1 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+))
  vm-imap-retrieve-messages((("420764" . 2) ("420793" . 3) ("420795" . 4) ("420797" . 5) ("420799" . 6) ("420803" . 7) ("420805" . 8) ("420807" . 9) ("420811" . 10) ("420813" . 11) ("420815" . 12) ("420817" . 13) ("420819" . 14) ("420831" . 15) ("420833" . 16) ("420835" . 17) ("420851" . 18) ("420859" . 19) ("420861" . 20) ("420863" . 21) ("420865" . 22) ("420873" . 23) ("420878" . 24) ("420880" . 25) ("420887" . 26) ("420917" . 27) ("420928" . 28) ("420968" . 29) ("420992" . 30) ("421010" . 31) ("421012" . 32) ("421019" . 33) ("421047" . 34) ("421053" . 35) ("421057" . 36) ("421066" . 37) ("421068" . 38) ("421092" . 39) ("421096" . 40) ("421190" . 41) ("421194" . 42) ("421202" . 43) ("421252" . 44) ("421428" . 45) ("421454" . 46) ("421458" . 47) ("421460" . 48) ("421462" . 49) ("421464" . 50) ("421472" . 51) ...))
  vm-imap-synchronize-folder(:interactive t :do-local-expunges t :do-retrieves t :save-attributes t :retrieve-attributes t)
  vm-get-spooled-mail(t)
  vm-get-new-mail(nil)
  funcall-interactively(vm-get-new-mail nil)
  command-execute(vm-get-new-mail)

I am forced to use Thunderbird or Office365 web Outlook. Maybe this bug is slightly blocking?

I suggest downloading the current vm trunk. If you want to avoid bzr, go to

  https://bazaar.launchpad.net/~vm/vm/trunk/files

click on "view revision", then "download tarball".

The only difference from a public release is that you need to do autoconf.
The INSTALL file gives complete instructions.

Cheers,
Uday

I did it, but again I got

Debugger entered--Lisp error: (wrong-type-argument char-or-string-p nil)
  vm-imap-retrieve-to-target(#<process IMAP over SSL> #<buffer inbox> [[nil 25276 537 577332 2 vm-imap-report-retrieval-status (#3) nil 509000 nil] t "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 575 nil nil "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 575 8192 87762] (IMAP4REV1 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+))
  vm-imap-retrieve-messages((("420764" . 2) ("420793" . 3) ("420795" . 4) ("420797" . 5) ("420799" . 6) ("420803" . 7) ("420805" . 8) ("420807" . 9) ("420811" . 10) ("420813" . 11) ("420815" . 12) ("420817" . 13) ("420819" . 14) ("420831" . 15) ("420833" . 16) ("420835" . 17) ("420851" . 18) ("420859" . 19) ("420861" . 20) ("420863" . 21) ("420865" . 22) ("420873" . 23) ("420878" . 24) ("420880" . 25) ("420887" . 26) ("420917" . 27) ("420928" . 28) ("420968" . 29) ("420992" . 30) ("421010" . 31) ("421012" . 32) ("421019" . 33) ("421047" . 34) ("421053" . 35) ("421057" . 36) ("421066" . 37) ("421068" . 38) ("421092" . 39) ("421096" . 40) ("421190" . 41) ("421194" . 42) ("421202" . 43) ("421252" . 44) ("421428" . 45) ("421454" . 46) ("421458" . 47) ("421460" . 48) ("421462" . 49) ("421464" . 50) ("421472" . 51) ...))
  vm-imap-synchronize-folder(:interactive t :do-local-expunges t :do-retrieves t :save-attributes t :retrieve-attributes t)
  vm-get-spooled-mail(t)
  vm-get-new-mail(nil)
  funcall-interactively(vm-get-new-mail nil)
  command-execute(vm-get-new-mail)

I attach the autoconf, configure and compile output

I noticed that the compiler output is almost boring ;) in recalling that package cl is deprecated and fires a lot of warnig like

[a vm-source-file].el:233:8: Warning: the function ‘[a named function]’ might not be defined at runtime.

Hi Uday,

VM no longer works with emacs 28 due to the JIT.

I have it at least partially working in:

  bzr+ssh://<email address hidden>/~markd-kermodei/vm/vm-emacs-28/

bzr is effectively dead; if VM is going to survive, it needs to be move to git,
and have some life infused into it.

try disabling native complication on vm files

Customize group Comp
Native Comp Deferred Compilation Deny List: vm-*.el

Disabled native compilation(*), removed vm-* files from native compilation cache, I restarted Emacs, called vm and got (I presume)

Debugger entered--Lisp error: (wrong-type-argument char-or-string-p nil)
  vm-imap-retrieve-to-target(#<process IMAP over SSL<1>> #<buffer inbox> [[nil 25283 63594 115317 2 vm-imap-report-retrieval-status (#3) nil 24000 nil] t "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 693 nil nil "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 693 12288 87762] (IMAP4REV1 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+))
  vm-imap-retrieve-messages((("420764" . 2) ("420793" . 3) ("420795" . 4) ("420797" . 5) ("420799" . 6) ("420803" . 7) ("420805" . 8) ("420807" . 9) ("420811" . 10) ("420813" . 11) ("420815" . 12) ("420817" . 13) ("420819" . 14) ("420831" . 15) ("420833" . 16) ("420835" . 17) ("420851" . 18) ("420859" . 19) ("420861" . 20) ("420863" . 21) ("420865" . 22) ("420873" . 23) ("420878" . 24) ("420880" . 25) ("420887" . 26) ("420917" . 27) ("420928" . 28) ("420968" . 29) ("420992" . 30) ("421010" . 31) ("421012" . 32) ("421019" . 33) ("421047" . 34) ("421053" . 35) ("421057" . 36) ("421066" . 37) ("421068" . 38) ("421092" . 39) ("421096" . 40) ("421190" . 41) ("421194" . 42) ("421202" . 43) ("421252" . 44) ("421428" . 45) ("421454" . 46) ("421458" . 47) ("421460" . 48) ("421462" . 49) ("421464" . 50) ("421472" . 51) ...))
  vm-imap-synchronize-folder(:interactive nil :do-local-expunges t :do-retrieves t :save-attributes t :retrieve-attributes t)
  vm-get-spooled-mail(nil)
  vm(nil :read-only nil)
  funcall-interactively(vm nil :read-only nil)
  command-execute(vm record)
  execute-extended-command(nil "vm" "vm")
  funcall-interactively(execute-extended-command nil "vm" "vm")
  command-execute(execute-extended-command)

I can only presume because the first stack trace disappeared automaticlally and I got this calling vm again with M-x vm.

(*) had to put a set on the top ov .emacs and change the regexp to "vm-.*\\.el"

I don't get this error from imap, so I am not sure what to
suggest. char-or-string-p isn't called directly in vm-imap.el,
so I would guess it is coming in from a macro.

I am not skilled enough with the elisp debugger to guide you
on finding out where the nil is coming from.

Gian Uberto Lauri <email address hidden> writes:
> Disabled native compilation(*), removed vm-* files from native
> compilation cache, I restarted Emacs, called vm and got (I presume)
>
> Debugger entered--Lisp error: (wrong-type-argument char-or-string-p nil)
> vm-imap-retrieve-to-target(#<process IMAP over SSL<1>> #<buffer inbox> [[nil 25283 63594 115317 2 vm-imap-report-retrieval-status (#3) nil 24000 nil] t "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 693 nil nil "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 693 12288 87762] (IMAP4REV1 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+))
> vm-imap-retrieve-messages((("420764" . 2) ("420793" . 3) ("420795" . 4) ("420797" . 5) ("420799" . 6) ("420803" . 7) ("420805" . 8) ("420807" . 9) ("420811" . 10) ("420813" . 11) ("420815" . 12) ("420817" . 13) ("420819" . 14) ("420831" . 15) ("420833" . 16) ("420835" . 17) ("420851" . 18) ("420859" . 19) ("420861" . 20) ("420863" . 21) ("420865" . 22) ("420873" . 23) ("420878" . 24) ("420880" . 25) ("420887" . 26) ("420917" . 27) ("420928" . 28) ("420968" . 29) ("420992" . 30) ("421010" . 31) ("421012" . 32) ("421019" . 33) ("421047" . 34) ("421053" . 35) ("421057" . 36) ("421066" . 37) ("421068" . 38) ("421092" . 39) ("421096" . 40) ("421190" . 41) ("421194" . 42) ("421202" . 43) ("421252" . 44) ("421428" . 45) ("421454" . 46) ("421458" . 47) ("421460" . 48) ("421462" . 49) ("421464" . 50) ("421472" . 51) ...))
> vm-imap-synchronize-folder(:interactive nil :do-local-expunges t :do-retrieves t :save-attributes t :retrieve-attributes t)
> vm-get-spooled-mail(nil)
> vm(nil :read-only nil)
> funcall-interactively(vm nil :read-only nil)
> command-execute(vm record)
> execute-extended-command(nil "vm" "vm")
> funcall-interactively(execute-extended-command nil "vm" "vm")
> command-execute(execute-extended-command)
>
> I can only presume because the first stack trace disappeared
> automaticlally and I got this calling vm again with M-x vm.
>
> (*) had to put a set on the top ov .emacs and change the regexp to
> "vm-.*\\.el"
>
> --

Unfortunately I don't have Emacs 29 yet. So I can't reproduce the problem.

Gian, after loading vm normally, please load vm-imap.el (the uncompiled
file) and post the backtrace.

(The compiled code doesn't tell us where exactly the problem happened.)

Uday

Thank you Uday!

I did something mode drastic, removed all compiled files from VM, sorry, it was a bit faster.

Now the stack trace is:

Debugger entered--Lisp error: (wrong-type-argument char-or-string-p nil)
  insert(nil)
  (let ((***start vm-imap-read-point) end fetch-response list p) (goto-char ***start) (vm-set-imap-status-got statblob 0) (let* ((func #'(lambda (beg end len) (if vm-imap-read-point (progn ... ...)))) (after-change-functions (cons func after-change-functions)) (need-ok t) response) (setq response (vm-imap-read-response-and-verify process "message FETCH")) (cond ((vm-imap-response-matches response '* 'atom 'FETCH 'list) (setq fetch-response response)) (t (vm-imap-normal-error "cannot retrieve message from the server")))) (setq vm-imap-read-point (point-marker)) (setq list (cdr (nth 3 fetch-response))) (cond (bodypeek (cond ((vm-imap-response-matches list 'BODY '(vector) 'string) (setq p (nth 2 list) ***start (nth 1 p))) ((vm-imap-response-matches list 'UID 'atom 'BODY '(vector) 'string) (setq p (nth 4 list) ***start (nth 1 p))) (t (vm-imap-protocol-error "expected (BODY[] string) in FETCH response")))) (t (if (not (vm-imap-response-matches list 'RFC822 'string)) (vm-imap-protocol-error "expected (RFC822 string) in FETCH response")) (setq p (nth 1 list) ***start (nth 1 p)))) (goto-char (nth 2 p)) (setq end (point-marker)) (vm-set-imap-status-need statblob nil) (vm-imap-cleanup-region ***start end) (vm-munge-message-separators vm-folder-type ***start end) (goto-char ***start) (vm-set-imap-status-got statblob nil) (if (and (eq vm-folder-type 'babyl) (cond ((stringp target) (let ((attrs ...)) (or (null attrs) (equal 0 ...)))) ((bufferp target) (save-current-buffer (set-buffer target) (= 0 (buffer-size)))))) (let ((opoint (point))) (vm-convert-folder-header nil vm-folder-type) (setq ***start opoint) (goto-char ***start) (vm-skip-past-folder-header))) (insert (vm-leading-message-separator)) (save-restriction (narrow-to-region (point) end) (vm-convert-folder-type-headers 'baremessage vm-folder-type)) (goto-char end) (if (and (not (eq 10 (char-after (1- (point))))) (memq vm-folder-type '(From_-with-Content-Length BellFrom_ From_))) (insert-before-markers "\n")) (insert-before-markers (vm-trailing-message-separator)) (if (stringp target) (let ((buffer-file-type t) (selective-display nil)) (write-region ***start end target t 0)) (let ((b (current-buffer))) (save-current-buffer (set-buffer target) (vm-buffer-type:enter 'unknown) (let ((buffer-read-only nil)) (insert-buffer-substring b ***start end)) (vm-buffer-type:exit)))) (delete-region ***start end) t)
  vm-imap-retrieve-to-target(#<process IMAP over SSL> #<buffer inbox> [[nil 25286 39507 441256 2 vm-imap-report-retrieval-status (#3) nil 152000 nil] t "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 753 nil nil "<email address hidden>:bbmxscphtwwpsgsr:inbox" 0 753 15352 87762] (IMAP4REV1 SASL-IR UIDPLUS ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+))
  (while (> k 0) (vm-imap-retrieve-to-target process folder-buffer statblob use-body-peek) (save-current-buffer (set-buffer folder-buffer) (if (= (point) pos) (debug "IMAP internal error #2012: the point hasn't moved"))) (setq k (1- ...

It is messy indeed on the web page

Bad news.

Microsoft removed the app password so I need oauth2

First of all thank you for your efforts

OK, DavMail solves the oauth2 issues, but after picking all my 1012 backlog message and having the code not natively compiled fixes everything. I will byte compile the code now :)

Wow, I can finally read the mail the way I like

even if I am forced to use no byte-compiled file for vm :(

@markd-kernmodie -- Thanks for the changes to get vm working with v28. They seem to be working for me.

In case anyone else is copying from here: Native Comp Deferred Compilation Deny List contains regexps, not globs, so the pattern would be "/vm-.*.el". At least that was what worked for me with Emacs 28.

Emacs 29 was finally released on 2023-07-30, so perhaps now would be a
good time to fix the problems? :-)

Some background. The roof starting caving in for me late last year when
Debian testing brought in the new Emacs 28.2 packages, configure to use
native compilation by default. Fortunately, after a few days of
desperation I found the advices to (selectively) inhibit native
compilation, and could read and write emails again.

These problems were also "fixed" (by disabling native compilation for
VM) in the Debian VM package following

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039105

only to re-surface again recently as a bloEmacs 29 was finally released on 2023-07-30, so perhaps now would be a
good time to fix the problems? :-)

Some background. The roof starting caving in for me late last year when
Debian testing brought in the new Emacs 28.2 packages, configure to use
native compilation by default. Fortunately, after a few days of
desperation I found the advices to (selectively) inhibit native
compilation, and could read and write emails again.

These problems were also "fixed" (by disabling native compilation for
VM) in the Debian VM package following

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039105

only to re-surface again recently as a blocker for getting Emacs 29 into
Debian, see

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1042889

(again, same "fix", but Emacs 29 had renamed the variable for disabling
native compilation).

So for the time being things should be "safe" in the sense that when
Emacs 29 makes it into Debian, VM will still work for me, but of course
ideally VM would be made to work with native compilation.

Perhaps Mark's changes could be merged into the VM master sources?cker for getting Emacs 29 into
Debian, see

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1042889

(again, same "fix", but Emacs 29 had renamed the variable for disabling
native compilation).

So for the time being things should be "safe" in the sense that when
Emacs 29 makes it into Debian, VM will still work for me, but of course
ideally VM would be made to work with native compilation.

Perhaps Mark's changes could be merged into the VM master sources?

Thank you Kurt for pointing to the relevand Debian bug. I've now updated the Fedora package similarly.

Is there anyone with commit rights to the repository still active? The VM trunk is not working with the two most recent major versions of Emacs. Mark has a branch above with fixes, and this and other reports discuss the issues, but there doesn't seem to happen anything.