/
opt
/
cppython
/
doc
/
ftputil
/
up file
home
<?xml version="1.0" encoding="utf-8" ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="generator" content="Docutils 0.16: http://docutils.sourceforge.net/" /> <title>ftputil -- a high-level FTP client library</title> <meta name="date" content="2021-03-18" /> <meta name="author" content="Stefan Schwarzer <sschwarzer@sschwarzer.net>" /> <style type="text/css"> /* :Author: David Goodger :Contact: goodger@users.sourceforge.net :Date: $Date: 2005-05-26 12:51:39 +0200 (Thu, 26 May 2005) $ :Version: $Revision: 3368 $ :Copyright: This stylesheet has been placed in the public domain. Default cascading style sheet for the HTML output of Docutils. */ /* "! important" is used here to override other ``margin-top`` and ``margin-bottom`` styles that are later in the stylesheet or more specific. See http://www.w3.org/TR/CSS1#the-cascade */ .first { margin-top: 0 ! important } .last, .with-subtitle { margin-bottom: 0 ! important } .hidden { display: none } a.toc-backref { text-decoration: none ; color: black } blockquote.epigraph { margin: 2em 5em ; } dl.docutils dd { margin-bottom: 0.5em } /* Uncomment (and remove this text!) to get bold-faced definition list terms dl.docutils dt { font-weight: bold } */ div.abstract { margin: 2em 5em } div.abstract p.topic-title { font-weight: bold ; text-align: center } div.admonition, div.attention, div.caution, div.danger, div.error, div.hint, div.important, div.note, div.tip, div.warning { margin: 2em ; border: medium outset ; padding: 1em } div.admonition p.admonition-title, div.hint p.admonition-title, div.important p.admonition-title, div.note p.admonition-title, div.tip p.admonition-title { font-weight: bold ; font-family: sans-serif } div.attention p.admonition-title, div.caution p.admonition-title, div.danger p.admonition-title, div.error p.admonition-title, div.warning p.admonition-title { color: red ; font-weight: bold ; font-family: sans-serif } /* Uncomment (and remove this text!) to get reduced vertical space in compound paragraphs. div.compound .compound-first, div.compound .compound-middle { margin-bottom: 0.5em } div.compound .compound-last, div.compound .compound-middle { margin-top: 0.5em } */ div.dedication { margin: 2em 5em ; text-align: center ; font-style: italic } div.dedication p.topic-title { font-weight: bold ; font-style: normal } div.figure { margin-left: 2em } div.footer, div.header { font-size: smaller } div.line-block { display: block ; margin-top: 1em ; margin-bottom: 1em } div.line-block div.line-block { margin-top: 0 ; margin-bottom: 0 ; margin-left: 1.5em } div.sidebar { margin-left: 1em ; border: medium outset ; padding: 1em ; background-color: #ffffee ; width: 40% ; float: right ; clear: right } div.sidebar p.rubric { font-family: sans-serif ; font-size: medium } div.system-messages { margin: 5em } div.system-messages h1 { color: red } div.system-message { border: medium outset ; padding: 1em } div.system-message p.system-message-title { color: red ; font-weight: bold } div.topic { margin: 2em } h1.section-subtitle, h2.section-subtitle, h3.section-subtitle, h4.section-subtitle, h5.section-subtitle, h6.section-subtitle { margin-top: 0.4em } h1.title { text-align: center } h2.subtitle { text-align: center } hr.docutils { width: 75% } ol.simple, ul.simple { margin-bottom: 1em } ol.arabic { list-style: decimal } ol.loweralpha { list-style: lower-alpha } ol.upperalpha { list-style: upper-alpha } ol.lowerroman { list-style: lower-roman } ol.upperroman { list-style: upper-roman } p.attribution { text-align: right ; margin-left: 50% } p.caption { font-style: italic } p.credits { font-style: italic ; font-size: smaller } p.label { white-space: nowrap } p.rubric { font-weight: bold ; font-size: larger ; color: maroon ; text-align: center } p.sidebar-title { font-family: sans-serif ; font-weight: bold ; font-size: larger } p.sidebar-subtitle { font-family: sans-serif ; font-weight: bold } p.topic-title { font-weight: bold } pre.address { margin-bottom: 0 ; margin-top: 0 ; font-family: serif ; font-size: 100% } pre.line-block { font-family: serif ; font-size: 100% } pre.literal-block, pre.doctest-block { margin-left: 2em ; margin-right: 2em ; background-color: #eeeeee } span.classifier { font-family: sans-serif ; font-style: oblique } span.classifier-delimiter { font-family: sans-serif ; font-weight: bold } span.interpreted { font-family: sans-serif } span.option { white-space: nowrap } span.pre { white-space: pre } span.problematic { color: red } span.section-subtitle { /* font-size relative to parent (<h#> element) */ font-size: 80% } table.citation { border-left: solid thin gray } table.docinfo { margin: 2em 4em } table.docutils { margin-top: 0.5em ; margin-bottom: 0.5em } table.footnote { border-left: solid thin black } table.docutils td, table.docutils th, table.docinfo td, table.docinfo th { padding-left: 0.5em ; padding-right: 0.5em ; vertical-align: top } table.docutils th.field-name, table.docinfo th.docinfo-name { font-weight: bold ; text-align: left ; white-space: nowrap ; padding-left: 0 } h1 tt.docutils, h2 tt.docutils, h3 tt.docutils, h4 tt.docutils, h5 tt.docutils, h6 tt.docutils { font-size: 100% } tt.docutils { background-color: #eeeeee } ul.auto-toc { list-style-type: none } </style> </head> <body> <div class="document" id="ftputil-a-high-level-ftp-client-library"> <h1 class="title"><tt class="docutils literal">ftputil</tt> -- a high-level FTP client library</h1> <table class="docinfo" frame="void" rules="none"> <col class="docinfo-name" /> <col class="docinfo-content" /> <tbody valign="top"> <tr><th class="docinfo-name">Version:</th> <td>5.0.1</td></tr> <tr><th class="docinfo-name">Date:</th> <td>2021-03-18</td></tr> <tr class="summary field"><th class="docinfo-name">Summary:</th><td class="field-body">high-level FTP client library for Python</td> </tr> <tr class="keywords field"><th class="docinfo-name">Keywords:</th><td class="field-body">FTP, <tt class="docutils literal">ftplib</tt> substitute, virtual filesystem, pure Python</td> </tr> <tr><th class="docinfo-name">Author:</th> <td>Stefan Schwarzer <<a class="reference external" href="mailto:sschwarzer@sschwarzer.net">sschwarzer@sschwarzer.net</a>></td></tr> </tbody> </table> <div class="contents topic" id="contents"> <p class="topic-title">Contents</p> <ul class="simple"> <li><a class="reference internal" href="#introduction" id="id4">Introduction</a></li> <li><a class="reference internal" href="#ftputil-features" id="id5"><tt class="docutils literal">ftputil</tt> features</a></li> <li><a class="reference internal" href="#exception-hierarchy" id="id6">Exception hierarchy</a></li> <li><a class="reference internal" href="#ftphost-objects" id="id7"><tt class="docutils literal">FTPHost</tt> objects</a><ul> <li><a class="reference internal" href="#construction" id="id8">Construction</a><ul> <li><a class="reference internal" href="#id1" id="id9">Introduction</a></li> <li><a class="reference internal" href="#session-factories" id="id10">Session factories</a></li> </ul> </li> <li><a class="reference internal" href="#directory-and-file-names" id="id11">Directory and file names</a></li> <li><a class="reference internal" href="#hidden-files-and-directories" id="id12">Hidden files and directories</a></li> <li><a class="reference internal" href="#ftphost-attributes-and-methods" id="id13"><tt class="docutils literal">FTPHost</tt> attributes and methods</a><ul> <li><a class="reference internal" href="#attributes" id="id14">Attributes</a></li> <li><a class="reference internal" href="#remote-file-system-navigation" id="id15">Remote file system navigation</a></li> <li><a class="reference internal" href="#uploading-and-downloading-files" id="id16">Uploading and downloading files</a></li> <li><a class="reference internal" href="#id2" id="id17">Time zone correction</a></li> <li><a class="reference internal" href="#creating-and-removing-directories" id="id18">Creating and removing directories</a></li> <li><a class="reference internal" href="#removing-files-and-links" id="id19">Removing files and links</a></li> <li><a class="reference internal" href="#retrieving-information-about-directories-files-and-links" id="id20">Retrieving information about directories, files and links</a></li> <li><a class="reference internal" href="#local-caching-of-file-system-information" id="id21">Local caching of file system information</a></li> <li><a class="reference internal" href="#iteration-over-directories" id="id22">Iteration over directories</a></li> <li><a class="reference internal" href="#other-methods" id="id23">Other methods</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#file-like-objects" id="id24">File-like objects</a><ul> <li><a class="reference internal" href="#id3" id="id25">Construction</a><ul> <li><a class="reference internal" href="#basics" id="id26">Basics</a></li> </ul> </li> <li><a class="reference internal" href="#attributes-and-methods" id="id27">Attributes and methods</a></li> </ul> </li> <li><a class="reference internal" href="#ftphost-instances-vs-ftp-connections" id="id28"><tt class="docutils literal">FTPHost</tt> instances vs. FTP connections</a></li> <li><a class="reference internal" href="#writing-directory-parsers" id="id29">Writing directory parsers</a></li> <li><a class="reference internal" href="#faq-tips-and-tricks" id="id30">FAQ / Tips and tricks</a><ul> <li><a class="reference internal" href="#where-can-i-get-the-latest-version" id="id31">Where can I get the latest version?</a></li> <li><a class="reference internal" href="#is-there-a-mailing-list-on-ftputil" id="id32">Is there a mailing list on <tt class="docutils literal">ftputil</tt>?</a></li> <li><a class="reference internal" href="#i-found-a-bug-what-now" id="id33">I found a bug! What now?</a></li> <li><a class="reference internal" href="#does-ftputil-support-tls" id="id34">Does <tt class="docutils literal">ftputil</tt> support TLS?</a></li> <li><a class="reference internal" href="#how-do-i-connect-to-a-non-default-port" id="id35">How do I connect to a non-default port?</a></li> <li><a class="reference internal" href="#how-do-i-set-active-or-passive-mode" id="id36">How do I set active or passive mode?</a></li> <li><a class="reference internal" href="#how-can-i-debug-an-ftp-connection-problem" id="id37">How can I debug an FTP connection problem?</a></li> <li><a class="reference internal" href="#conditional-upload-download-to-from-a-server-in-a-different-time-zone" id="id38">Conditional upload/download to/from a server in a different time zone</a></li> <li><a class="reference internal" href="#when-i-use-ftputil-all-i-get-is-a-parsererror-exception" id="id39">When I use <tt class="docutils literal">ftputil</tt>, all I get is a <tt class="docutils literal">ParserError</tt> exception</a></li> <li><a class="reference internal" href="#isdir-isfile-or-islink-incorrectly-return-false" id="id40"><tt class="docutils literal">isdir</tt>, <tt class="docutils literal">isfile</tt> or <tt class="docutils literal">islink</tt> incorrectly return <tt class="docutils literal">False</tt></a></li> <li><a class="reference internal" href="#i-don-t-find-an-answer-to-my-problem-in-this-document" id="id41">I don't find an answer to my problem in this document</a></li> </ul> </li> <li><a class="reference internal" href="#bugs-and-limitations" id="id42">Bugs and limitations</a></li> <li><a class="reference internal" href="#files" id="id43">Files</a></li> <li><a class="reference internal" href="#references" id="id44">References</a></li> <li><a class="reference internal" href="#authors" id="id45">Authors</a></li> </ul> </div> <div class="section" id="introduction"> <h1><a class="toc-backref" href="#id4">Introduction</a></h1> <p>The <tt class="docutils literal">ftputil</tt> module is a high-level interface to the <a class="reference external" href="https://docs.python.org/library/ftplib.html">ftplib</a> module. The <a class="reference internal" href="#ftphost-objects">FTPHost objects</a> generated from it allow many operations similar to those of <a class="reference external" href="https://docs.python.org/library/os.html">os</a>, <a class="reference external" href="https://docs.python.org/library/os.path.html">os.path</a> and <a class="reference external" href="https://docs.python.org/library/shutil.html">shutil</a>.</p> <p>Example:</p> <pre class="literal-block"> import ftputil # Download some files from the login directory. with ftputil.FTPHost("ftp.domain.com", "user", "password") as ftp_host: names = ftp_host.listdir(ftp_host.curdir) for name in names: if ftp_host.path.isfile(name): ftp_host.download(name, name) # remote, local # Make a new directory and copy a remote file into it. ftp_host.mkdir("newdir") with ftp_host.open("index.html", "rb") as source: with ftp_host.open("newdir/index.html", "wb") as target: ftp_host.copyfileobj(source, target) # similar to shutil.copyfileobj </pre> <p>Also, there are <a class="reference internal" href="#ftphost-lstat">FTPHost.lstat</a> and <a class="reference internal" href="#ftphost-stat">FTPHost.stat</a> to request size and modification time of a file. The latter can also follow links, similar to <a class="reference external" href="https://docs.python.org/library/os.html#os.stat">os.stat</a>. <a class="reference internal" href="#ftphost-walk">FTPHost.walk</a> and <a class="reference internal" href="#ftphost-path-walk">FTPHost.path.walk</a> work, too.</p> </div> <div class="section" id="ftputil-features"> <h1><a class="toc-backref" href="#id5"><tt class="docutils literal">ftputil</tt> features</a></h1> <ul class="simple"> <li>Method names are familiar from Python's <tt class="docutils literal">os</tt>, <tt class="docutils literal">os.path</tt> and <tt class="docutils literal">shutil</tt> modules. For example, use <tt class="docutils literal">os.path.join</tt> to join paths for a local file system and <tt class="docutils literal">ftp_host.path.join</tt> to join paths for a remote FTP file system.</li> <li>Remote file system navigation (<tt class="docutils literal">getcwd</tt>, <tt class="docutils literal">chdir</tt>)</li> <li>Upload and download files (<tt class="docutils literal">upload</tt>, <tt class="docutils literal">upload_if_newer</tt>, <tt class="docutils literal">download</tt>, <tt class="docutils literal">download_if_newer</tt>)</li> <li>Time zone synchronization between client and server (needed for <tt class="docutils literal">upload_if_newer</tt> and <tt class="docutils literal">download_if_newer</tt>)</li> <li>Create and remove directories (<tt class="docutils literal">mkdir</tt>, <tt class="docutils literal">makedirs</tt>, <tt class="docutils literal">rmdir</tt>, <tt class="docutils literal">rmtree</tt>) and remove files (<tt class="docutils literal">remove</tt>)</li> <li>Get information about directories, files and links (<tt class="docutils literal">listdir</tt>, <tt class="docutils literal">stat</tt>, <tt class="docutils literal">lstat</tt>, <tt class="docutils literal">exists</tt>, <tt class="docutils literal">isdir</tt>, <tt class="docutils literal">isfile</tt>, <tt class="docutils literal">islink</tt>, <tt class="docutils literal">abspath</tt>, <tt class="docutils literal">dirname</tt>, <tt class="docutils literal">basename</tt> etc.)</li> <li>Iterate over remote file systems (<tt class="docutils literal">walk</tt>)</li> <li>Local caching of results from <tt class="docutils literal">lstat</tt> and <tt class="docutils literal">stat</tt> calls to reduce network access (also applies to <tt class="docutils literal">exists</tt>, <tt class="docutils literal">getmtime</tt> etc.).</li> <li>Read files from and write files to remote hosts via file-like objects (<tt class="docutils literal">FTPHost.open</tt>; the generated file-like objects have the familiar methods like <tt class="docutils literal">read</tt>, <tt class="docutils literal">readline</tt>, <tt class="docutils literal">readlines</tt>, <tt class="docutils literal">write</tt>, <tt class="docutils literal">writelines</tt> and <tt class="docutils literal">close</tt>. You can also iterate over these files line by line in a <tt class="docutils literal">for</tt> loop.</li> </ul> </div> <div class="section" id="exception-hierarchy"> <h1><a class="toc-backref" href="#id6">Exception hierarchy</a></h1> <p>The exceptions are in the namespace of the <tt class="docutils literal">ftputil.error</tt> module, e. g. <tt class="docutils literal">ftputil.error.TemporaryError</tt>.</p> <p>The exception classes are organized as follows:</p> <pre class="literal-block"> FTPError FTPOSError(FTPError, OSError) PermanentError(FTPOSError) CommandNotImplementedError(PermanentError) TemporaryError(FTPOSError) FTPIOError(FTPError) InternalError(FTPError) InaccessibleLoginDirError(InternalError) NoEncodingError(InternalError) ParserError(InternalError) RootDirError(InternalError) TimeShiftError(InternalError) </pre> <p>and are described here:</p> <ul> <li><p class="first"><tt class="docutils literal">FTPError</tt></p> <p>is the root of the exception hierarchy of the module.</p> </li> <li><p class="first"><tt class="docutils literal">FTPOSError</tt></p> <p>is derived from <tt class="docutils literal">OSError</tt>. This is for similarity between the os module and <tt class="docutils literal">FTPHost</tt> objects. Compare</p> <pre class="literal-block"> try: os.chdir("nonexisting_directory") except OSError: ... </pre> <p>with</p> <pre class="literal-block"> host = ftputil.FTPHost("host", "user", "password") try: host.chdir("nonexisting_directory") except OSError: ... </pre> <p>Imagine a function</p> <pre class="literal-block"> def func(path, file): ... </pre> <p>which works on the local file system and catches <tt class="docutils literal">OSErrors</tt>. If you change the parameter list to</p> <pre class="literal-block"> def func(path, file, os=os): ... </pre> <p>where <tt class="docutils literal">os</tt> denotes the <tt class="docutils literal">os</tt> module, you can call the function also as</p> <pre class="literal-block"> host = ftputil.FTPHost("host", "user", "password") func(path, file, os=host) </pre> <p>to use the same code for both a local and remote file system. Another similarity between <tt class="docutils literal">OSError</tt> and <tt class="docutils literal">FTPOSError</tt> is that the latter holds the FTP server return code in the <tt class="docutils literal">errno</tt> attribute of the exception object and the error text in <tt class="docutils literal">strerror</tt>.</p> </li> <li><p class="first"><tt class="docutils literal">PermanentError</tt></p> <p>is raised for 5xx return codes from the FTP server. This corresponds to <tt class="docutils literal">ftplib.error_perm</tt> (though <tt class="docutils literal">PermanentError</tt> and <tt class="docutils literal">ftplib.error_perm</tt> are <em>not</em> identical).</p> </li> <li><p class="first"><tt class="docutils literal">CommandNotImplementedError</tt></p> <p>indicates that an underlying command the code tries to use is not implemented. For an example, see the description of the <a class="reference internal" href="#ftphost-chmod">FTPHost.chmod</a> method.</p> </li> <li><p class="first"><tt class="docutils literal">TemporaryError</tt></p> <p>is raised for FTP return codes from the 4xx category. This corresponds to <tt class="docutils literal">ftplib.error_temp</tt> (though <tt class="docutils literal">TemporaryError</tt> and <tt class="docutils literal">ftplib.error_temp</tt> are <em>not</em> identical).</p> </li> <li><p class="first"><tt class="docutils literal">FTPIOError</tt></p> <p>denotes an I/O error on the remote host. This appears mainly with file-like objects that are retrieved by calling <tt class="docutils literal">FTPHost.open</tt>. Compare</p> <pre class="literal-block"> >>> try: ... f = open("not_there") ... except IOError as obj: ... print(obj.errno) ... print(obj.strerror) ... 2 No such file or directory </pre> <p>with</p> <pre class="literal-block"> >>> ftp_host = ftputil.FTPHost("host", "user", "password") >>> try: ... f = ftp_host.open("not_there") ... except IOError as obj: ... print(obj.errno) ... print(obj.strerror) ... 550 550 not_there: No such file or directory. </pre> <p>As you can see, both code snippets are similar. However, the error codes aren't the same.</p> </li> <li><p class="first"><tt class="docutils literal">InternalError</tt></p> <p>subsumes exception classes for signaling errors due to limitations of the FTP protocol or the concrete implementation of <tt class="docutils literal">ftputil</tt>.</p> </li> <li><p class="first"><tt class="docutils literal">InaccessibleLoginDirError</tt></p> <p>This exception is raised if the directory in which "you" are placed upon login is not accessible, i. e. a <tt class="docutils literal">chdir</tt> call with the directory as argument would fail.</p> </li> <li><p class="first"><tt class="docutils literal">NoEncodingError</tt></p> <p>is raised if an FTP session instance doesn't have an <tt class="docutils literal">encoding</tt> attribute (see also <a class="reference internal" href="#session-factories">session factories</a>).</p> </li> <li><p class="first"><tt class="docutils literal">ParserError</tt></p> <p>is used for errors during the parsing of directory listings from the server. This exception is used by the <tt class="docutils literal">FTPHost</tt> methods <tt class="docutils literal">stat</tt>, <tt class="docutils literal">lstat</tt>, and <tt class="docutils literal">listdir</tt>.</p> </li> <li><p class="first"><tt class="docutils literal">RootDirError</tt></p> <p>Because of the implementation of the <tt class="docutils literal">lstat</tt> method it is not possible to do a <tt class="docutils literal">stat</tt> call on the root directory <tt class="docutils literal">/</tt>. If you know <em>any</em> way to do it, please let me know. :-)</p> <p>This problem does <em>not</em> affect stat calls on items <em>in</em> the root directory.</p> </li> <li><p class="first"><tt class="docutils literal">TimeShiftError</tt></p> <p>is used to denote errors which relate to setting the <a class="reference internal" href="#time-shift">time shift</a>.</p> </li> </ul> </div> <div class="section" id="ftphost-objects"> <h1><a class="toc-backref" href="#id7"><tt class="docutils literal">FTPHost</tt> objects</a></h1> <div class="section" id="construction"> <span id="ftphost-construction"></span><h2><a class="toc-backref" href="#id8">Construction</a></h2> <div class="section" id="id1"> <h3><a class="toc-backref" href="#id9">Introduction</a></h3> <p><tt class="docutils literal">FTPHost</tt> instances can be created with the following call:</p> <pre class="literal-block"> ftp_host = ftputil.FTPHost(server, user, password, account, session_factory=ftplib.FTP) </pre> <p>The first four parameters are strings with the same meaning as for the FTP class in the <tt class="docutils literal">ftplib</tt> module. Usually the <tt class="docutils literal">account</tt> and <tt class="docutils literal">session_factory</tt> arguments aren't needed though.</p> <p><tt class="docutils literal">FTPHost</tt> objects can also be used in a <tt class="docutils literal">with</tt> statement:</p> <pre class="literal-block"> import ftputil with ftputil.FTPHost(server, user, password) as ftp_host: print(ftp_host.listdir(ftp_host.curdir)) </pre> <p>After the <tt class="docutils literal">with</tt> block, the <tt class="docutils literal">FTPHost</tt> instance and the associated FTP sessions will be closed automatically.</p> <p>If something goes wrong during the <tt class="docutils literal">FTPHost</tt> construction or in the body of the <tt class="docutils literal">with</tt> statement, the instance is closed as well. Exceptions will be propagated (as with <tt class="docutils literal">try ... finally</tt>).</p> </div> <div class="section" id="session-factories"> <span id="session-factory"></span><h3><a class="toc-backref" href="#id10">Session factories</a></h3> <p>The keyword argument <tt class="docutils literal">session_factory</tt> may be used to generate FTP connections with other factories than the default <tt class="docutils literal">ftplib.FTP</tt>. For example, the standard library of Python 3 contains a class <tt class="docutils literal">ftplib.FTP_TLS</tt> which extends <tt class="docutils literal">ftplib.FTP</tt> to use an encrypted connection.</p> <p>In fact, all positional and keyword arguments other than <tt class="docutils literal">session_factory</tt> are passed to the factory to generate a new background session. This also happens for every remote file that is opened; see below.</p> <p>This functionality of the constructor also allows to wrap <tt class="docutils literal">ftplib.FTP</tt> objects to do something that wouldn't be possible with the <tt class="docutils literal">ftplib.FTP</tt> constructor alone.</p> <p>As an example, assume you want to connect to another than the default port, but <tt class="docutils literal">ftplib.FTP</tt> only offers this by means of its <tt class="docutils literal">connect</tt> method, not via its constructor. One solution is to use a custom class as a session factory:</p> <pre class="literal-block"> import ftplib import ftputil EXAMPLE_PORT = 50001 class MySession(ftplib.FTP): def __init__(self, host, userid, password, port): """Act like ftplib.FTP's constructor but connect to another port.""" ftplib.FTP.__init__(self) self.connect(host, port) self.login(userid, password) # Try _not_ to use an _instance_ `MySession()` as factory, - # use the class itself. with ftputil.FTPHost(host, userid, password, port=EXAMPLE_PORT, session_factory=MySession) as ftp_host: # Use `ftp_host` as usual. ... </pre> <p>On login, the format of the directory listings (needed for stat'ing files and directories) should be determined automatically. If not, please <a class="reference external" href="https://ftputil.sschwarzer.net/issuetrackernotes">enter a ticket</a>.</p> <p>For the most common uses you don't need to create your own session factory class though. The <tt class="docutils literal">ftputil.session</tt> module has a function <tt class="docutils literal">session_factory</tt> that can create session factories for a variety of parameters:</p> <pre class="literal-block"> session_factory(base_class=ftplib.FTP, port=21, use_passive_mode=None, encrypt_data_channel=True, encoding=None, debug_level=None) </pre> <p>with</p> <ul> <li><p class="first"><tt class="docutils literal">base_class</tt> is a base class to inherit a new session factory class from. By default, this is <tt class="docutils literal">ftplib.FTP</tt> from the Python standard library.</p> </li> <li><p class="first"><tt class="docutils literal">port</tt> is the command channel port. The default is 21, used in most FTP server configurations.</p> </li> <li><p class="first"><tt class="docutils literal">use_passive_mode</tt> is either a boolean that determines whether passive mode should be used or <tt class="docutils literal">None</tt>. <tt class="docutils literal">None</tt> means to let the base class choose active or passive mode.</p> </li> <li><p class="first"><tt class="docutils literal">encrypt_data_channel</tt> defines whether to encrypt the data channel for secure connections. This is only supported for the base classes <tt class="docutils literal">ftplib.FTP_TLS</tt> and <tt class="docutils literal">M2Crypto.ftpslib.FTP_TLS</tt>, otherwise the parameter is ignored.</p> </li> <li><p class="first"><tt class="docutils literal">encoding</tt> can be a string to set the encoding of directory and file paths on the remote server. (This has nothing to do with the encoding of file contents!) If you pass a string and your base class is neither <tt class="docutils literal">ftplib.FTP</tt> nor <tt class="docutils literal">ftplib.FTP_TLS</tt>, the used heuristic in <tt class="docutils literal">session_factory</tt> may not work reliably. Therefore, if in doubt, let <tt class="docutils literal">encoding</tt> be <tt class="docutils literal">None</tt> and define your <tt class="docutils literal">base_class</tt> so that it sets the encoding you want.</p> <p>Note: In Python 3.9, the default path encoding for <tt class="docutils literal">ftplib.FTP</tt> and <tt class="docutils literal">ftplib.FTP_TLS</tt> changed from previously "latin-1" to "utf-8". Hence, if you don't pass an <tt class="docutils literal">encoding</tt> to <tt class="docutils literal">session_factory</tt>, you'll get different path encodings for Python 3.8 and earlier vs. Python 3.9 and later.</p> <p>If you're sure that you always use only ASCII characters in your remote paths, you don't need to worry about the path encoding and don't need to use the <tt class="docutils literal">encoding</tt> argument.</p> </li> <li><p class="first"><tt class="docutils literal">debug_level</tt> sets the debug level for FTP session instances. The semantics is defined by the base class. For example, a debug level of 2 causes the most verbose output for Python's <tt class="docutils literal">ftplib.FTP</tt> class.</p> </li> </ul> <p>All of these parameters can be combined. For example, you could use</p> <pre class="literal-block"> import ftplib import ftputil import ftputil.session my_session_factory = ftputil.session.session_factory( base_class=ftpslib.FTP_TLS, port=31, encrypt_data_channel=True, encoding="UTF-8", debug_level=2) with ftputil.FTPHost(server, user, password, session_factory=my_session_factory) as ftp_host: ... </pre> <p>to create and use a session factory derived from <tt class="docutils literal">ftplib.FTP_TLS</tt> that connects on command channel 31, will encrypt the data channel, use the UTF-8 encoding for remote paths and print output for debug level 2.</p> <p>Note: Generally, you can achieve everything you can do with <tt class="docutils literal">ftputil.session.session_factory</tt> with an explicit session factory as described at the start of this section.</p> </div> </div> <div class="section" id="directory-and-file-names"> <h2><a class="toc-backref" href="#id11">Directory and file names</a></h2> <div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">Keep in mind that this section only applies to directory and file <em>names</em>, not file <em>contents</em>. Encoding and decoding for file contents is handled by the <tt class="docutils literal">encoding</tt> argument for <a class="reference internal" href="#ftphost-open">FTPHost.open</a>.</p> </div> <p>Generally, paths can be <tt class="docutils literal">str</tt> or <tt class="docutils literal">bytes</tt> objects (or <a class="reference external" href="https://docs.python.org/3/library/os.html#os.PathLike">PathLike</a> objects wrapping <tt class="docutils literal">str</tt> or <tt class="docutils literal">bytes</tt>). However, you can't mix different string types (<tt class="docutils literal">bytes</tt> and <tt class="docutils literal">str</tt>) in one call (for example in <tt class="docutils literal">FTPHost.path.join</tt>). If a method gets a string argument (or a string argument wrapped in a <a class="reference external" href="https://docs.python.org/3/library/os.html#os.PathLike">PathLike</a> object) and returns one or more strings, these strings will have the same string type (<tt class="docutils literal">bytes</tt> or <tt class="docutils literal">str</tt>) as the argument(s). Mixing different string types in one call (for example in <tt class="docutils literal">FTPHost.path.join</tt>) isn't allowed and will cause a <tt class="docutils literal">TypeError</tt>. These rules are the same as for local file system operations.</p> <p>Although you can pass paths as <tt class="docutils literal">str</tt> or <tt class="docutils literal">bytes</tt>, the former is recommended. See below for the reason.</p> <p><em>If</em> you have directory or file names with non-ASCII characters, you need to be aware of the encoding the <a class="reference internal" href="#session-factory">session factory</a> (e. g. <tt class="docutils literal">ftplib.FTP</tt>) uses. This needs to be the same encoding that the FTP server uses for the paths.</p> <p>The following diagram shows string conversions on the way from your code to the remote FTP server. The opposite way works analogously, so encoding steps in the diagram become decoding steps and decoding steps in the diagram become encoding steps.</p> <p>Both "branching points" in the upper and lower part of diagrams are independent, so depending on how you pass paths to ftputil and which file system API the FTP server uses, there are four possible combinations.</p> <pre class="literal-block"> +-----------+ +-----------+ | Your code | | Your code | +-----------+ +-----------+ | | | str | bytes v v +-------------+ +-------------+ decode with encoding of session, | ftputil API | | ftputil API | e. g. `ftplib.FTP` instance +-------------+ +-------------+ \ / \ str / v v +---------------+ encode with encoding | ftplib API | specified in `FTP` instance +---------------+ | | bytes v +-------------+ | socket API | +-------------+ / \ / \ local / client - - - - - / - - - - - \ - - - - - - - - - - - - - - - - - - - - - - / \ remote / server / bytes \ v v +------------+ +------------+ decode with encoding from | FTP server | | FTP server | FTP server configuration +------------+ +------------+ | | | bytes | str v v +-------------+ +-------------+ | remote file | | remote file | | system API | | system API | +-------------+ +-------------+ \ / \ bytes / v v +-------------------+ | file system | +-------------------+ </pre> <p>As you can see at the top of the diagram, if you use <tt class="docutils literal">str</tt> objects (regular unicode strings), there's one fewer decoding step, and so one fewer source of problems. If you use <tt class="docutils literal">bytes</tt> objects for paths, ftputil tries to get the encoding for the FTP server from the <tt class="docutils literal">encoding</tt> attribute of the session instance (say, an instance of <tt class="docutils literal">ftplib.FTP</tt>). If no <tt class="docutils literal">encoding</tt> attribute is present, a <tt class="docutils literal">NoEncodingError</tt> is raised.</p> <p>All encoding/decoding steps must use the same encoding, the encoding the server uses (at the bottom of the diagram). If the server uses the bytes from the socket directly, i. e. without an encoding step, you have to use the file system encoding.</p> <p>Until and including Python 3.8, the encoding implicitly assumed by the <tt class="docutils literal">ftplib</tt> module was latin-1, so using <tt class="docutils literal">bytes</tt> was the safest strategy. However, Python 3.9 made the <tt class="docutils literal">encoding</tt> configurable via an <tt class="docutils literal">ftplib.FTP</tt> constructor argument <tt class="docutils literal">encoding</tt>, <em>but defaults to UTF-8</em>.</p> <p>If you don't pass a <a class="reference internal" href="#session-factory">session factory</a> to the <tt class="docutils literal">ftputil.FTPHost</tt> constructor, ftputil will use latin-1 encoding for the paths. This is the same value as in earlier ftputil versions in combination with Python 3.8 and earlier.</p> <p>Summary:</p> <ul class="simple"> <li>If possible, use only ASCII characters in paths.</li> <li>If possible, pass paths to ftputil as <tt class="docutils literal">str</tt>, not <tt class="docutils literal">bytes</tt>.</li> <li>If you use a custom session factory, the session instances created by the factory must have an <tt class="docutils literal">encoding</tt> attribute with the name of the path encoding to use. If your session instances don't have an <tt class="docutils literal">encoding</tt> attribute, ftputil raises a <tt class="docutils literal">NoEncodingError</tt> when the session is created.</li> </ul> </div> <div class="section" id="hidden-files-and-directories"> <h2><a class="toc-backref" href="#id12">Hidden files and directories</a></h2> <p>Whether ftputil sees "hidden" files and directories (usually files or directories whose names start with a dot) depends on the FTP server configuration. By default, ftputil does <em>not</em> use the <tt class="docutils literal"><span class="pre">-a</span></tt> option in the FTP <tt class="docutils literal">LIST</tt> command to find hidden files.</p> <p>To tell the server to list hidden directories and files, set <tt class="docutils literal">FTPHost.use_list_a_option</tt> to <tt class="docutils literal">True</tt>:</p> <pre class="literal-block"> ftp_host = ftputil.FTPHost(server, user, password, account, session_factory=ftplib.FTP) ftp_host.use_list_a_option = True </pre> <p>Caveats:</p> <ul class="simple"> <li>If the server doesn't understand the <tt class="docutils literal"><span class="pre">-a</span></tt> option at all, the server may interpret <tt class="docutils literal"><span class="pre">-a</span></tt> as the name of a file or directory, which can result in odd behavior. Therefore, use <tt class="docutils literal"><span class="pre">-a</span></tt> only if you're sure the server you're talking to supports it. Another approach is to have test code for <tt class="docutils literal"><span class="pre">-a</span></tt> support and fall back to not using the option.</li> <li>Even if the server knows about the <tt class="docutils literal"><span class="pre">-a</span></tt> option, the server may be configured to ignore it.</li> </ul> </div> <div class="section" id="ftphost-attributes-and-methods"> <h2><a class="toc-backref" href="#id13"><tt class="docutils literal">FTPHost</tt> attributes and methods</a></h2> <div class="section" id="attributes"> <h3><a class="toc-backref" href="#id14">Attributes</a></h3> <ul> <li><p class="first"><tt class="docutils literal">curdir</tt>, <tt class="docutils literal">pardir</tt>, <tt class="docutils literal">sep</tt></p> <p>are strings which denote the current and the parent directory on the remote server. <tt class="docutils literal">sep</tt> holds the path separator. Though <a class="reference external" href="https://www.ietf.org/rfc/rfc959.txt">RFC 959</a> (File Transfer Protocol) notes that these values may depend on the FTP server implementation, the Unix variants seem to work well in practice, even for non-Unix servers.</p> <p>Nevertheless, it's recommended that you don't hardcode these values for remote paths, but use <a class="reference internal" href="#ftphost-path">FTPHost.path</a> as you would use <tt class="docutils literal">os.path</tt> to write platform-independent Python code for local filesystems. Keep in mind that most, <em>but not all</em>, arguments of <tt class="docutils literal">FTPHost</tt> methods refer to remote directories or files. For example, in <a class="reference internal" href="#uploading-and-downloading-files">FTPHost.upload</a>, the first argument is a local path and the second a remote path. Both of these should use their respective path separators.</p> </li> </ul> </div> <div class="section" id="remote-file-system-navigation"> <h3><a class="toc-backref" href="#id15">Remote file system navigation</a></h3> <ul> <li><p class="first"><tt class="docutils literal">getcwd()</tt></p> <p>returns the absolute current directory on the remote host. This method works like <tt class="docutils literal">os.getcwd</tt>.</p> </li> <li><p class="first"><tt class="docutils literal">chdir(directory)</tt></p> <p>sets the current directory on the FTP server. This resembles <tt class="docutils literal">os.chdir</tt>, as you may have expected.</p> </li> </ul> </div> <div class="section" id="uploading-and-downloading-files"> <span id="callback-function"></span><h3><a class="toc-backref" href="#id16">Uploading and downloading files</a></h3> <ul> <li><p class="first"><tt class="docutils literal">upload(source, target, callback=None)</tt></p> <p>copies a local source file (given by a filename, i. e. a string) to the remote host under the name target. Both <tt class="docutils literal">source</tt> and <tt class="docutils literal">target</tt> may be absolute paths or relative to their corresponding current directory (on the local or the remote host, respectively).</p> <p>The file content is always transferred in binary mode.</p> <p>The callback, if given, will be invoked for each transferred chunk of data:</p> <pre class="literal-block"> callback(chunk) </pre> <p>where <tt class="docutils literal">chunk</tt> is a bytestring. An example usage of a callback method is to display a progress indicator.</p> </li> <li><p class="first"><tt class="docutils literal">download(source, target, callback=None)</tt></p> <p>performs a download from the remote source file to a local target file. Both <tt class="docutils literal">source</tt> and <tt class="docutils literal">target</tt> are strings. See the description of <tt class="docutils literal">upload</tt> for more details.</p> </li> </ul> <ul id="upload-if-newer"> <li><p class="first"><tt class="docutils literal">upload_if_newer(source, target, callback=None)</tt></p> <p>is similar to the <tt class="docutils literal">upload</tt> method. The only difference is that the upload is only invoked if the time of the last modification for the source file is more recent than that of the target file or the target doesn't exist at all. The check for the last modification time considers the precision of the timestamps and transfers a file "if in doubt". Consequently the code</p> <pre class="literal-block"> ftp_host.upload_if_newer("source_file", "target_file") time.sleep(10) ftp_host.upload_if_newer("source_file", "target_file") </pre> <p>might upload the file again if the timestamp of the target file is precise up to a minute, which is typically the case because the remote datetime is determined by parsing a directory listing from the server. To avoid unnecessary transfers, wait at least a minute between calls of <tt class="docutils literal">upload_if_newer</tt> for the same file. If it still seems that a file is uploaded unnecessarily (or not when it should), read the subsection on <a class="reference internal" href="#time-shift">time shift</a> settings.</p> <p>If an upload actually happened, the return value of <tt class="docutils literal">upload_if_newer</tt> is a <tt class="docutils literal">True</tt>, else <tt class="docutils literal">False</tt>.</p> <p>Note that the method only checks the existence and/or the modification time of the source and target file; it doesn't compare any other file properties, say, the file size.</p> <p>This also means that if a transfer is interrupted, the remote file will have a newer modification time than the local file, and thus the transfer won't be repeated if <tt class="docutils literal">upload_if_newer</tt> is used a second time. There are at least two possibilities after a failed upload:</p> <ul class="simple"> <li>use <tt class="docutils literal">upload</tt> instead of <tt class="docutils literal">upload_if_newer</tt>, or</li> <li>remove the incomplete target file with <tt class="docutils literal">FTPHost.remove</tt>, then use <tt class="docutils literal">upload</tt> or <tt class="docutils literal">upload_if_newer</tt> to transfer it again.</li> </ul> </li> </ul> <ul id="download-if-newer"> <li><p class="first"><tt class="docutils literal">download_if_newer(source, target, callback=None)</tt></p> <p>corresponds to <tt class="docutils literal">upload_if_newer</tt> but performs a download from the server to the local host. Read the descriptions of download and <tt class="docutils literal">upload_if_newer</tt> for more information. If a download actually happened, the return value is <tt class="docutils literal">True</tt>, else <tt class="docutils literal">False</tt>.</p> </li> </ul> </div> <div class="section" id="id2"> <span id="time-zone-correction"></span><span id="time-shift"></span><h3><a class="toc-backref" href="#id17">Time zone correction</a></h3> <p>For <tt class="docutils literal">upload_if_newer</tt> and <tt class="docutils literal">download_if_newer</tt> to work correctly, the time zone of the server must be taken into account. By default, ftputil assumes that the timestamps in server listings are in <a class="reference external" href="https://en.wikipedia.org/wiki/Utc">UTC</a>.</p> <ul id="set-time-shift"> <li><p class="first"><tt class="docutils literal">set_time_shift(time_shift)</tt></p> <p>sets the so-called time shift value, measured in seconds. The time shift here is defined as the difference between the time used in server listings and UTC.</p> <pre class="literal-block"> time_shift = server_time - utc_time </pre> <p>For example, a server in Berlin/Germany set to the local time (currently UTC+03:00), would require a time shift value of 3 * 3600.0 = 10800.0 seconds to be handled correctly by ftputil's <tt class="docutils literal">upload_if_newer</tt> and <tt class="docutils literal">download_if_newer</tt>, as well as the <tt class="docutils literal">stat</tt> and <tt class="docutils literal">lstat</tt> calls.</p> <p>Note that servers don't necessarily send their file system listings in their local time zone. Some use UTC, which actually makes sense because UTC doesn't lead to an ambiguity when there's a switch back from the daylight saving time to the "normal" time of the server location.</p> <p>If the time shift value is invalid, for example its absolute value is larger than 24 hours, a <tt class="docutils literal">TimeShiftError</tt> is raised.</p> <div class="admonition note"> <p class="first admonition-title">Note</p> <p>Versions of ftputil before 4.0.0 used a different definition of "time shift", server_time - local_client_time.</p> <p class="last">This had the advantage that the default of 0.0 would be correct <em>if</em> the server was set to the same time zone as the client where ftputil runs. On the other hand, this approach meant that the time shift depended on <em>two</em> time zones, not only the one used on the server side. This could be confusing if server and client <em>didn't</em> use the same time zone.</p> </div> <p>See also <a class="reference internal" href="#synchronize-times">synchronize_times</a> for a way to set the time shift with a simple method call. If you can't use <tt class="docutils literal">synchronize_times</tt> <em>and</em> the server uses the same time zone as the client, you can set the time shift value with</p> <pre class="literal-block"> set_time_shift( round( (datetime.datetime.now() - datetime.datetime.utcnow()).seconds, -2 ) ) </pre> </li> <li><p class="first"><tt class="docutils literal">time_shift()</tt></p> <p>returns the currently-set time shift value. See <tt class="docutils literal">set_time_shift</tt> above for its definition.</p> </li> </ul> <ul id="synchronize-times"> <li><p class="first"><tt class="docutils literal">synchronize_times()</tt></p> <p>synchronizes the local times of the server and the client, so that <a class="reference internal" href="#upload-if-newer">upload_if_newer</a> and <a class="reference internal" href="#download-if-newer">download_if_newer</a> work as expected, even if the client and the server use different time zones. For this to work, <em>all</em> of the following conditions must be true:</p> <ul class="simple"> <li>The connection between server and client is established.</li> <li>The client has write access to the directory that is current when <tt class="docutils literal">synchronize_times</tt> is called.</li> </ul> <p>If you can't fulfill these conditions, you can nevertheless set the time shift value explicitly with <a class="reference internal" href="#set-time-shift">set_time_shift</a>. Trying to call <tt class="docutils literal">synchronize_times</tt> if the above conditions aren't met results in a <tt class="docutils literal">TimeShiftError</tt> exception.</p> </li> </ul> </div> <div class="section" id="creating-and-removing-directories"> <h3><a class="toc-backref" href="#id18">Creating and removing directories</a></h3> <ul> <li><p class="first"><tt class="docutils literal">mkdir(path, [mode])</tt></p> <p>makes the given directory on the remote host. This does <em>not</em> construct "intermediate" directories that don't already exist. The <tt class="docutils literal">mode</tt> parameter is ignored; this is for compatibility with <tt class="docutils literal">os.mkdir</tt> if an <tt class="docutils literal">FTPHost</tt> object is passed into a function instead of the <tt class="docutils literal">os</tt> module. See the explanation in the subsection <a class="reference internal" href="#exception-hierarchy">Exception hierarchy</a>.</p> </li> <li><p class="first"><tt class="docutils literal">makedirs(path, [mode], exist_ok=False)</tt></p> <p>works similar to <tt class="docutils literal">mkdir</tt> (see above), but also makes intermediate directories like <tt class="docutils literal">os.makedirs</tt>. The <tt class="docutils literal">mode</tt> parameter is only there for compatibility with <tt class="docutils literal">os.makedirs</tt> and is ignored.</p> <p><tt class="docutils literal">exist_ok</tt> controls whether the existence of any directory but the last in the <tt class="docutils literal">path</tt> should be considered an error. If the default <tt class="docutils literal">False</tt> is used or passed to <tt class="docutils literal">makedirs</tt>, ftputil will raise a <tt class="docutils literal">PermanentError</tt> if any directory but the last already exists.</p> </li> <li><p class="first"><tt class="docutils literal">rmdir(path)</tt></p> <p>removes the given remote directory. If it's not empty, raise a <tt class="docutils literal">PermanentError</tt>.</p> </li> <li><p class="first"><tt class="docutils literal">rmtree(path, ignore_errors=False, onerror=None)</tt></p> <p>removes the given remote, possibly non-empty, directory tree. The interface of this method is rather complex, in favor of compatibility with <tt class="docutils literal">shutil.rmtree</tt>.</p> <p>If <tt class="docutils literal">ignore_errors</tt> is set to a true value, errors are ignored. If <tt class="docutils literal">ignore_errors</tt> is a false value <em>and</em> <tt class="docutils literal">onerror</tt> isn't set, all exceptions occurring during the tree iteration and processing are raised. These exceptions are all of type <tt class="docutils literal">PermanentError</tt>.</p> <p>To distinguish between different kinds of errors, pass in a callable for <tt class="docutils literal">onerror</tt>. This callable must accept three arguments: <tt class="docutils literal">func</tt>, <tt class="docutils literal">path</tt> and <tt class="docutils literal">exc_info</tt>. <tt class="docutils literal">func</tt> is a bound method object, <em>for example</em> <tt class="docutils literal">your_host_object.listdir</tt>. <tt class="docutils literal">path</tt> is the path that was the recent argument of the respective method (<tt class="docutils literal">listdir</tt>, <tt class="docutils literal">remove</tt>, <tt class="docutils literal">rmdir</tt>). <tt class="docutils literal">exc_info</tt> is the exception info as it is gotten from <tt class="docutils literal">sys.exc_info</tt>.</p> <p>The code of <tt class="docutils literal">rmtree</tt> is taken from Python's <tt class="docutils literal">shutil</tt> module and adapted for <tt class="docutils literal">ftputil</tt>.</p> </li> </ul> </div> <div class="section" id="removing-files-and-links"> <h3><a class="toc-backref" href="#id19">Removing files and links</a></h3> <ul> <li><p class="first"><tt class="docutils literal">remove(path)</tt></p> <p>removes a file or link on the remote host, similar to <tt class="docutils literal">os.remove</tt>.</p> </li> <li><p class="first"><tt class="docutils literal">unlink(path)</tt></p> <p>is an alias for <tt class="docutils literal">remove</tt>.</p> </li> </ul> </div> <div class="section" id="retrieving-information-about-directories-files-and-links"> <h3><a class="toc-backref" href="#id20">Retrieving information about directories, files and links</a></h3> <ul> <li><p class="first"><tt class="docutils literal">listdir(path)</tt></p> <p>returns a list containing the names of the files and directories in the given path, similar to <a class="reference external" href="https://docs.python.org/library/os.html#os.listdir">os.listdir</a>. The special names <tt class="docutils literal">.</tt> and <tt class="docutils literal">..</tt> are not in the list.</p> </li> </ul> <p>The methods <tt class="docutils literal">lstat</tt> and <tt class="docutils literal">stat</tt> (and some others) rely on the directory listing format used by the FTP server. When connecting to a host, <tt class="docutils literal">FTPHost</tt>'s constructor tries to guess the right format, which succeeds in most cases. However, if you get strange results or <tt class="docutils literal">ParserError</tt> exceptions by a mere <tt class="docutils literal">lstat</tt> call, please <a class="reference external" href="https://ftputil.sschwarzer.net/issuetrackernotes">enter a ticket</a>.</p> <p>If <tt class="docutils literal">lstat</tt> or <tt class="docutils literal">stat</tt> give wrong modification dates or times, look at the methods that deal with time zone differences (<a class="reference internal" href="#time-zone-correction">time zone correction</a>).</p> <ul id="ftphost-lstat"> <li><p class="first"><tt class="docutils literal">lstat(path)</tt></p> <p>returns an object similar to that from <a class="reference external" href="https://docs.python.org/library/os.html#os.lstat">os.lstat</a>. This is a kind of tuple with additional attributes; see the documentation of the <tt class="docutils literal">os</tt> module for details.</p> <p>The result is derived by parsing the output of a <tt class="docutils literal">LIST</tt> command on the server. Therefore, the result from <tt class="docutils literal">FTPHost.lstat</tt> can not contain more information than the received text. In particular:</p> <ul> <li><p class="first">User and group ids can only be determined as strings, not as numbers, and that only if the server supplies them. This is usually the case with Unix servers but maybe not for other FTP servers.</p> </li> <li><p class="first">Values for the time of the last modification may be rough, depending on the information from the server. For timestamps older than a year, this usually means that the precision of the modification timestamp value is not better than a day. For newer files, the information may be accurate to a minute.</p> <p>If the time of the last modification is before the epoch (usually 1970-01-01 UTC), set the time of the last modification to 0.0.</p> </li> <li><p class="first">Links can only be recognized on servers that provide this information in the <tt class="docutils literal">LIST</tt> output.</p> </li> <li><dl class="first docutils"> <dt>Stat attributes that can't be determined at all are set to</dt> <dd><p class="first last"><tt class="docutils literal">None</tt>. For example, a line of a directory listing may not contain the date/time of a directory's last modification.</p> </dd> </dl> </li> <li><p class="first">There's a special problem with stat'ing the root directory. (Stat'ing things <em>in</em> the root directory is fine though.) In this case, a <tt class="docutils literal">RootDirError</tt> is raised. This has to do with the algorithm used by <tt class="docutils literal">(l)stat</tt>, and I know of no approach which mends this problem.</p> </li> </ul> <p>Currently, <tt class="docutils literal">ftputil</tt> recognizes the common Unix-style and Microsoft/DOS-style directory formats. If you need to parse output from another server type, please write to the <a class="reference external" href="https://ftputil.sschwarzer.net/mailinglist">ftputil mailing list</a>. You may consider <a class="reference internal" href="#writing-directory-parsers">writing your own parser</a>.</p> </li> </ul> <ul id="ftphost-stat"> <li><p class="first"><tt class="docutils literal">stat(path)</tt></p> <p>returns <tt class="docutils literal">stat</tt> information also for files which are pointed to by a link. This method follows multiple links until a regular file or directory is found. If an infinite link chain is encountered or the target of the last link in the chain doesn't exist, a <tt class="docutils literal">PermanentError</tt> is raised.</p> <p>The limitations of the <tt class="docutils literal">lstat</tt> method also apply to <tt class="docutils literal">stat</tt>.</p> </li> </ul> <p id="ftphost-path"><tt class="docutils literal">FTPHost</tt> objects contain an attribute named <tt class="docutils literal">path</tt>, similar to <a class="reference external" href="https://docs.python.org/library/os.path.html">os.path</a>. The following methods can be applied to the remote host with the same semantics as for <tt class="docutils literal">os.path</tt>:</p> <pre class="literal-block"> abspath(path) basename(path) commonprefix(path_list) dirname(path) exists(path) getmtime(path) getsize(path) isabs(path) isdir(path) isfile(path) islink(path) join(path1, path2, ...) normcase(path) normpath(path) split(path) splitdrive(path) splitext(path) walk(path, func, arg) </pre> <p>Like Python's counterparts under <a class="reference external" href="https://docs.python.org/library/os.path.html">os.path</a>, <tt class="docutils literal">ftputil</tt>'s <tt class="docutils literal"><span class="pre">is...</span></tt> methods return <tt class="docutils literal">False</tt> if they can't find the path given by their argument.</p> </div> <div class="section" id="local-caching-of-file-system-information"> <h3><a class="toc-backref" href="#id21">Local caching of file system information</a></h3> <p>Many of the above methods need access to the remote file system to obtain data on directories and files. To get the most recent data, <em>each</em> call to <tt class="docutils literal">lstat</tt>, <tt class="docutils literal">stat</tt>, <tt class="docutils literal">exists</tt>, <tt class="docutils literal">getmtime</tt> etc. would require to fetch a directory listing from the server, which can make the program <em>very</em> slow. This effect is more pronounced for operations which mostly scan the file system rather than transferring file data.</p> <p>For this reason, <tt class="docutils literal">ftputil</tt> by default saves the results from directory listings locally and reuses those results. This reduces network accesses and so speeds up the software a lot. However, since data is more rarely fetched from the server, the risk of obsolete data also increases. This will be discussed below.</p> <p>Caching can be controlled -- if necessary at all -- via the <tt class="docutils literal">stat_cache</tt> object in an <tt class="docutils literal">FTPHost</tt>'s namespace. For example, after calling</p> <pre class="literal-block"> ftp_host = ftputil.FTPHost(host, user, password) </pre> <p>the cache can be accessed as <tt class="docutils literal">ftp_host.stat_cache</tt>.</p> <p>While <tt class="docutils literal">ftputil</tt> usually manages the cache quite well, there are two possible reasons for modifying cache parameters.</p> <p>The first is when the number of possible entries is too low. You may notice that when you are processing very large directories and the program becomes much slower than before. It's common for code to read a directory with <tt class="docutils literal">listdir</tt> and then process the found directories and files. This can also happen implicitly by a call to <tt class="docutils literal">FTPHost.walk</tt>. Since version 2.6 <tt class="docutils literal">ftputil</tt> automatically increases the cache size if directories with more entries than the current maximum cache size are to be scanned. Most of the time, this works fine.</p> <p>However, if you need access to stat data for several directories at the same time, you may need to increase the cache explicitly. This is done by the <tt class="docutils literal">resize</tt> method:</p> <pre class="literal-block"> ftp_host.stat_cache.resize(20000) </pre> <p>where the argument is the maximum number of <tt class="docutils literal">lstat</tt> results to store (the default is 5000, in versions before 2.6 it was 1000). Note that each path on the server, e. g. "/home/schwa/some_dir", corresponds to a single cache entry. Methods like <tt class="docutils literal">exists</tt> or <tt class="docutils literal">getmtime</tt> all derive their results from a previously fetched <tt class="docutils literal">lstat</tt> result.</p> <p>The value 5000 above means that the cache will hold <em>at most</em> 5000 entries (unless increased automatically by an explicit or implicit <tt class="docutils literal">listdir</tt> call, see above). If more are about to be stored, the entries which haven't been used for the longest time will be deleted to make place for newer entries.</p> <p>The second possible reason to change the cache parameters is to avoid stale cache data. Caching is so effective because it reduces network accesses. This can also be a disadvantage if the file system data on the remote server changes after a stat result has been retrieved; the client, when looking at the cached stat data, will use obsolete information.</p> <p>There are two potential ways to get such out-of-date stat data. The first happens when an <tt class="docutils literal">FTPHost</tt> instance modifies a file path for which it has a cache entry, e. g. by calling <tt class="docutils literal">remove</tt> or <tt class="docutils literal">rmdir</tt>. Such changes are handled transparently; the path will be deleted from the cache. A different matter are changes unknown to the <tt class="docutils literal">FTPHost</tt> object which inspects its cache. Obviously, for example, these are changes by programs running on the remote host. On the other hand, cache inconsistencies can also occur if two <tt class="docutils literal">FTPHost</tt> objects change a file system simultaneously:</p> <pre class="literal-block"> with ( ftputil.FTPHost(server, user1, password1) as ftp_host1, ftputil.FTPHost(server, user1, password1) as ftp_host2 ): stat_result1 = ftp_host1.stat("some_file") stat_result2 = ftp_host2.stat("some_file") ftp_host2.remove("some_file") # `ftp_host1` will still see the obsolete cache entry! print(ftp_host1.stat("some_file")) # Will raise an exception since an `FTPHost` object # knows of its own changes. print(ftp_host2.stat("some_file")) </pre> <p>At first sight, it may appear to be a good idea to have a shared cache among several <tt class="docutils literal">FTPHost</tt> objects. After some thinking, this turns out to be very error-prone. For example, it won't help with different processes using <tt class="docutils literal">ftputil</tt>. So, if you have to deal with concurrent write/read accesses to a server, you have to handle them explicitly.</p> <p>The most useful tool for this is the <tt class="docutils literal">invalidate</tt> method. In the example above, it could be used like this:</p> <pre class="literal-block"> with ( ftputil.FTPHost(server, user1, password1) as ftp_host1, ftputil.FTPHost(server, user1, password1) as ftp_host2 ): stat_result1 = ftp_host1.stat("some_file") stat_result2 = ftp_host2.stat("some_file") ftp_host2.remove("some_file") # Invalidate using an absolute path. absolute_path = ftp_host1.path.abspath( ftp_host1.path.join(ftp_host1.getcwd(), "some_file")) ftp_host1.stat_cache.invalidate(absolute_path) # Will now raise an exception as it should. print(ftp_host1.stat("some_file")) # Would raise an exception since an `FTPHost` object # knows of its own changes, even without `invalidate`. print(ftp_host2.stat("some_file")) </pre> <p>The method <tt class="docutils literal">invalidate</tt> can be used on any <em>absolute</em> path, be it a directory, a file or a link.</p> <p>By default, the cache entries (if not replaced by newer ones) are stored for an infinite time. That is, if you start your Python process using <tt class="docutils literal">ftputil</tt> and let it run for three days a stat call may still access cache data that old. To avoid this, you can set the <tt class="docutils literal">max_age</tt> attribute:</p> <pre class="literal-block"> with ftputil.FTPHost(server, user, password) as ftp_host: ftp_host.stat_cache.max_age = 60 * 60 # = 3600 seconds </pre> <p>This sets the maximum age of entries in the cache to an hour. This means any entry older won't be retrieved from the cache but its data instead fetched again from the remote host and then again stored for up to an hour. To reset <cite>max_age</cite> to the default of unlimited age, i. e. cache entries never expire, use <tt class="docutils literal">None</tt> as value.</p> <p>If you are certain that the cache will be in the way, you can disable and later re-enable it completely with <tt class="docutils literal">disable</tt> and <tt class="docutils literal">enable</tt>:</p> <pre class="literal-block"> with ftputil.FTPHost(server, user, password) as ftp_host: ftp_host.stat_cache.disable() ... ftp_host.stat_cache.enable() </pre> <p>During that time, the cache won't be used; all data will be fetched from the network. After enabling the cache again, its entries will be the same as when the cache was disabled, that is, entries won't get updated with newer data during this period. Note that even when the cache is disabled, the file system data in the code can become inconsistent:</p> <pre class="literal-block"> with ftputil.FTPHost(server, user, password) as ftp_host: ftp_host.stat_cache.disable() if ftp_host.path.exists("some_file"): mtime = ftp_host.path.getmtime("some_file") </pre> <p>In that case, the file <tt class="docutils literal">some_file</tt> may have been removed by another process between the calls to <tt class="docutils literal">exists</tt> and <tt class="docutils literal">getmtime</tt>!</p> </div> <div class="section" id="iteration-over-directories"> <h3><a class="toc-backref" href="#id22">Iteration over directories</a></h3> <ul id="ftphost-walk"> <li><p class="first"><tt class="docutils literal">walk(top, topdown=True, onerror=None, followlinks=False)</tt></p> <p>iterates over a directory tree, similar to <a class="reference external" href="https://docs.python.org/2/library/os.html#os.walk">os.walk</a>. Actually, <tt class="docutils literal">FTPHost.walk</tt> uses the code from Python with just the necessary modifications, so see the linked documentation.</p> </li> </ul> <ul id="ftphost-path-walk"> <li><p class="first"><tt class="docutils literal">path.walk(path, func, arg)</tt></p> <p>Similar to <tt class="docutils literal">os.path.walk</tt>, the <tt class="docutils literal">walk</tt> method in <a class="reference internal" href="#ftphost-path">FTPHost.path</a> can be used, though <tt class="docutils literal">FTPHost.walk</tt> is probably easier to use.</p> </li> </ul> </div> <div class="section" id="other-methods"> <h3><a class="toc-backref" href="#id23">Other methods</a></h3> <ul> <li><p class="first"><tt class="docutils literal">close()</tt></p> <p>closes the connection to the remote host. After this, no more interaction with the FTP server is possible with this <tt class="docutils literal">FTPHost</tt> object. Usually you don't need to close an <tt class="docutils literal">FTPHost</tt> instance with <tt class="docutils literal">close</tt> if you set up the instance in a <tt class="docutils literal">with</tt> statement.</p> </li> <li><p class="first"><tt class="docutils literal">rename(source, target)</tt></p> <p>renames the source file (or directory) on the FTP server.</p> </li> </ul> <ul id="ftphost-chmod"> <li><p class="first"><tt class="docutils literal">chmod(path, mode)</tt></p> <p>sets the access mode (permission flags) for the given path. The mode is an integer as returned for the mode by the <tt class="docutils literal">stat</tt> and <tt class="docutils literal">lstat</tt> methods. Be careful: Usually, mode values are written as octal numbers, for example 0755 to make a directory readable and writable for the owner, but not writable for the group and others. If you want to use such octal values, rely on Python's support for them:</p> <pre class="literal-block"> ftp_host.chmod("some_directory", 0o755) </pre> <p>Not all FTP servers support the <tt class="docutils literal">chmod</tt> command. In case of an exception, how do you know if the path doesn't exist or if the command itself is invalid? If the FTP server complies with <a class="reference external" href="https://www.ietf.org/rfc/rfc959.txt">RFC 959</a>, it should return a status code 502 if the <tt class="docutils literal">SITE CHMOD</tt> command isn't allowed. <tt class="docutils literal">ftputil</tt> maps this special error response to a <tt class="docutils literal">CommandNotImplementedError</tt> which is derived from <tt class="docutils literal">PermanentError</tt>.</p> <p>So you need to code like this:</p> <pre class="literal-block"> with ftputil.FTPHost(server, user, password) as ftp_host: try: ftp_host.chmod("some_file", 0o644) except ftputil.error.CommandNotImplementedError: # `chmod` not supported ... except ftputil.error.PermanentError: # Possibly a non-existent file ... </pre> <p>Because the <tt class="docutils literal">CommandNotImplementedError</tt> is more specific, you have to test for it first.</p> </li> </ul> <ul> <li><p class="first"><tt class="docutils literal">copyfileobj(source, target, length=64*1024)</tt></p> <p>copies the contents from the file-like object <tt class="docutils literal">source</tt> to the file-like object <tt class="docutils literal">target</tt>. The only difference to <tt class="docutils literal">shutil.copyfileobj</tt> is the default buffer size. Note that arbitrary file-like objects can be used as arguments (e. g. local files, remote FTP files).</p> <p>However, the interfaces of <tt class="docutils literal">source</tt> and <tt class="docutils literal">target</tt> have to match; the string type read from <tt class="docutils literal">source</tt> must be an accepted string type when written to <tt class="docutils literal">target</tt>. For example, if you open <tt class="docutils literal">source</tt> in Python 3 as a local text file and <tt class="docutils literal">target</tt> as a remote file object in binary mode, the transfer will fail since <tt class="docutils literal">source.read</tt> gives unicode strings (<tt class="docutils literal">str</tt>) whereas <tt class="docutils literal">target.write</tt> only accepts byte strings (<tt class="docutils literal">bytes</tt>).</p> <p>See <a class="reference internal" href="#file-like-objects">File-like objects</a> for the construction and use of remote file-like objects.</p> </li> </ul> <ul id="set-parser"> <li><p class="first"><tt class="docutils literal">set_parser(parser)</tt></p> <p>sets a custom parser for FTP directories. Note that you have to pass in a parser <em>instance</em>, not the class.</p> <p>An <a class="reference internal" href="#writing-directory-parsers">extra section</a> shows how to write own parsers if the default parsers in <tt class="docutils literal">ftputil</tt> don't work for you.</p> </li> </ul> <ul id="keep-alive"> <li><p class="first"><tt class="docutils literal">keep_alive()</tt></p> <p>attempts to keep the connection to the remote server active in order to prevent timeouts from happening. This method is primarily intended to keep the underlying FTP connection of an <tt class="docutils literal">FTPHost</tt> object alive while a file is uploaded or downloaded. This will require either an extra thread while the upload or download is in progress or calling <tt class="docutils literal">keep_alive</tt> from a <a class="reference internal" href="#callback-function">callback function</a>.</p> <p>The <tt class="docutils literal">keep_alive</tt> method won't help if the connection has already timed out. In this case, a <tt class="docutils literal">ftputil.error.TemporaryError</tt> is raised.</p> <p>If you want to use this method, keep in mind that FTP servers define a timeout for a reason. A timeout prevents running out of server connections because of clients that never disconnect on their own.</p> <p>Note that the <tt class="docutils literal">keep_alive</tt> method does <em>not</em> affect the "hidden" FTP child connections established by <tt class="docutils literal">FTPHost.open</tt> (see section <a class="reference internal" href="#ftphost-instances-vs-ftp-connections">FTPHost instances vs. FTP connections</a> for details). You <em>can't</em> use <tt class="docutils literal">keep_alive</tt> to avoid a timeout in a stalling transfer like this:</p> <pre class="literal-block"> with ftputil.FTPHost(server, userid, password) as ftp_host: with ftp_host.open("some_remote_file", "rb") as fobj: data = fobj.read(100) # _Futile_ attempt to avoid file connection timeout. for i in range(15): time.sleep(60) ftp_host.keep_alive() # Will raise an `ftputil.error.TemporaryError`. data += fobj.read() </pre> </li> </ul> </div> </div> </div> <div class="section" id="file-like-objects"> <span id="ftphost-open"></span><h1><a class="toc-backref" href="#id24">File-like objects</a></h1> <div class="section" id="id3"> <h2><a class="toc-backref" href="#id25">Construction</a></h2> <div class="section" id="basics"> <h3><a class="toc-backref" href="#id26">Basics</a></h3> <p><tt class="docutils literal">FTPFile</tt> objects are returned by a call to <tt class="docutils literal">FTPHost.open</tt>; never use the <tt class="docutils literal">FTPFile</tt> constructor directly.</p> <p>The APIs for remote file-like objects is modeled after the APIs of the built-in <tt class="docutils literal">open</tt> function and its return value.</p> <ul> <li><p class="first"><tt class="docutils literal">FTPHost.open(path, <span class="pre">mode="r",</span> buffering=None, encoding=None, errors=None, newline=None, rest=None)</tt></p> <p>returns a file-like object that refers to the path on the remote host. This path may be absolute or relative to the current directory on the remote host (this directory can be determined with the <tt class="docutils literal">getcwd</tt> method). As with local file objects, the default mode is "r", i. e. reading text files. Valid modes are "r", "rb", "w", and "wb".</p> <p>If a file is opened in binary mode, you <em>must not</em> specify an encoding. On the other hand, if you open a file in text mode, an encoding is used. By default, this is the return value of <tt class="docutils literal">locale.getpreferredencoding</tt>, but you can (and probably should) specify a distinct encoding.</p> <p>If you open a file in binary mode, the read and write operations use <tt class="docutils literal">bytes</tt> objects. That is, read operations return <tt class="docutils literal">bytes</tt> and write operations only accept <tt class="docutils literal">bytes</tt>.</p> <p>Similarly, text files always work with strings (<tt class="docutils literal">str</tt>). Here, read operations return string and write operations only accept strings.</p> <p>The arguments <tt class="docutils literal">buffering</tt>, <tt class="docutils literal">errors</tt> and <tt class="docutils literal">newline</tt> have the same semantics as in <a class="reference external" href="https://docs.python.org/3/library/functions.html#open">open</a>.</p> <p>If the file is opened in binary mode, you may pass 0 or a positive integer for the <tt class="docutils literal">rest</tt> argument. The argument is passed to the underlying FTP session instance (for example an instance of <tt class="docutils literal">ftplib.FTP</tt>) to start reading or writing at the given byte offset. For example, if a remote file contains the letters "abcdef" in ASCII encoding, <tt class="docutils literal">rest=3</tt> will start reading at "d".</p> <div class="admonition warning"> <p class="first admonition-title">Warning</p> <p class="last">If you pass <tt class="docutils literal">rest</tt> values which point <em>after</em> the file, the behavior is undefined and may even differ from one FTP server to another. Therefore, use the <tt class="docutils literal">rest</tt> argument only for error recovery in case of interrupted transfers. You need to keep track of the transferred data so that you can provide a valid <tt class="docutils literal">rest</tt> argument for a resumed transfer.</p> </div> </li> </ul> <p><tt class="docutils literal">FTPHost.open</tt> can also be used in a <tt class="docutils literal">with</tt> statement:</p> <pre class="literal-block"> import ftputil with ftputil.FTPHost(...) as ftp_host: ... with ftp_host.open("new_file", "w", encoding="utf8") as fobj: fobj.write("This is some text.") </pre> <p>At the end of the <tt class="docutils literal">with</tt> block, the remote file will be closed automatically.</p> <p>If something goes wrong during the construction of the file or in the body of the <tt class="docutils literal">with</tt> statement, the file will be closed as well. Exceptions will be propagated as with <tt class="docutils literal">try ... finally</tt>.</p> </div> </div> <div class="section" id="attributes-and-methods"> <h2><a class="toc-backref" href="#id27">Attributes and methods</a></h2> <p>The methods</p> <pre class="literal-block"> close() read([count]) readline([count]) readlines() write(data) writelines(string_sequence) </pre> <p>and the attribute <tt class="docutils literal">closed</tt> have the same semantics as for file objects of a local disk file system. The iterator protocol is supported as well, i. e. you can use a loop to read a file line by line:</p> <pre class="literal-block"> with ftputil.FTPHost(server, user, password) as ftp_host: with ftp_host.open("some_file") as input_file: for line in input_file: # Do something with the line, e. g. print(line.strip().replace("ftplib", "ftputil")) </pre> <p>For more on file objects, see the section <a class="reference external" href="https://docs.python.org/3/glossary.html#term-file-object">File objects</a> in the Python Library Reference.</p> </div> </div> <div class="section" id="ftphost-instances-vs-ftp-connections"> <span id="child-connections"></span><h1><a class="toc-backref" href="#id28"><tt class="docutils literal">FTPHost</tt> instances vs. FTP connections</a></h1> <p>This section explains why keeping an <tt class="docutils literal">FTPHost</tt> instance "alive" without timing out sometimes isn't trivial. If you always finish your FTP operations in time, you don't need to read this section.</p> <p>The file transfer protocol is a stateful protocol. That means an FTP connection always is in a certain state. Each of these states can only change to certain other states under certain conditions triggered by the client or the server.</p> <p>One of the consequences is that a single FTP connection can't be used at the same time, say, to transfer data on the FTP data channel and to create a directory on the remote host.</p> <p>For example, consider this:</p> <pre class="literal-block"> >>> import ftplib >>> ftp = ftplib.FTP(server, user, password) >>> ftp.pwd() '/' >>> # Start transfer. `CONTENTS` is a text file on the server. >>> socket = ftp.transfercmd("RETR CONTENTS") >>> socket <socket._socketobject object at 0x7f801a6386e0> >>> ftp.pwd() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/ftplib.py", line 578, in pwd return parse257(resp) File "/usr/lib64/python2.7/ftplib.py", line 842, in parse257 raise error_reply, resp ftplib.error_reply: 226-File successfully transferred 226 0.000 seconds (measured here), 5.60 Mbytes per second >>> </pre> <p>Note that <tt class="docutils literal">ftp</tt> is a single FTP connection, represented by an <tt class="docutils literal">ftplib.FTP</tt> instance, not an <tt class="docutils literal">ftputil.FTPHost</tt> instance.</p> <p>On the other hand, consider this:</p> <pre class="literal-block"> >>> import ftputil >>> ftp_host = ftputil.FTPHost(server, user, password) >>> ftp_host.getcwd() >>> fobj = ftp_host.open("CONTENTS") >>> fobj <ftputil.file.FTPFile object at 0x7f8019d3aa50> >>> ftp_host.getcwd() u'/' >>> fobj.readline() u'Contents of FTP test directory\n' >>> fobj.close() >>> </pre> <p>To be able to start a file transfer (i. e. open a remote file for reading or writing) and still be able to use other FTP commands, ftputil uses a trick. For every remote file, ftputil creates a new FTP connection, called a child connection in the ftputil source code. (Actually, FTP connections belonging to closed remote files are re-used if they haven't timed out yet.)</p> <p>In most cases this approach isn't noticeable by code using ftputil. However, the nice abstraction of dealing with a single FTP connection falls apart if one of the child connections times out. For example, if you open a remote file and work only with the initial "main" connection to navigate the file system, the FTP connection for the remote file may eventually time out.</p> <p>While it's often relatively easy to prevent the "main" connection from timing out it's unfortunately practically impossible to do this for a remote file connection (apart from transferring some data, of course). For this reason, <a class="reference internal" href="#keep-alive">FTPHost.keep_alive</a> affects only the main connection. Child connections may still time out if they're idle for too long.</p> <p>Some more details:</p> <ul class="simple"> <li>A kind of "straightforward" way of keeping the main connection alive would be to call <tt class="docutils literal">ftp_host.getcwd()</tt>. However, this doesn't work because ftputil caches the current directory and returns it without actually contacting the server. That's the main reason why there's a <tt class="docutils literal">keep_alive</tt> method since it calls <tt class="docutils literal">pwd</tt> on the FTP connection (i. e. the session object), which isn't a public attribute.</li> <li>Some servers define not only an idle timeout but also a transfer timeout. This means the connection times out unless there's some transfer on the data channel for this connection. So ftputil's <tt class="docutils literal">keep_alive</tt> doesn't prevent this timeout, but an <tt class="docutils literal">ftp_host.listdir(ftp_host.curdir)</tt> call should do it. However, this transfers the data for the whole directory listing which might take some time if the directory has many entries.</li> </ul> <p>Bottom line: If you can, you should organize your FTP actions so that you finish everything before a timeout happens.</p> </div> <div class="section" id="writing-directory-parsers"> <h1><a class="toc-backref" href="#id29">Writing directory parsers</a></h1> <p><tt class="docutils literal">ftputil</tt> recognizes the two most widely-used FTP directory formats, Unix and MS style, and adjusts itself automatically. Almost every FTP server uses one of these formats.</p> <p>However, if your server uses a format which is different from the two provided by <tt class="docutils literal">ftputil</tt>, you can plug in a custom parser with a single method call and have <tt class="docutils literal">ftputil</tt> use this parser.</p> <p>For this, you need to write a parser class by inheriting from the class <tt class="docutils literal">Parser</tt> in the <tt class="docutils literal">ftputil.stat</tt> module. Here's an example:</p> <pre class="literal-block"> import ftputil.error import ftputil.stat class XyzParser(ftputil.stat.Parser): """ Parse the default format of the FTP server of the XYZ corporation. """ def parse_line(self, line, time_shift=0.0): """ Parse a `line` from the directory listing and return a corresponding `StatResult` object. If the line can't be parsed, raise `ftputil.error.ParserError`. The `time_shift` argument can be used to fine-tune the parsing of dates and times. See the class `ftputil.stat.UnixParser` for an example. """ # Split the `line` argument and examine it further; if # something goes wrong, raise an `ftputil.error.ParserError`. ... # Make a `StatResult` object from the parts above. stat_result = ftputil.stat.StatResult(...) # `_st_name`, `_st_target` and `_st_mtime_precision` are optional. stat_result._st_name = ... stat_result._st_target = ... stat_result._st_mtime_precision = ... return stat_result # Define `ignores_line` only if the default in the base class # doesn't do enough! def ignores_line(self, line): """ Return a true value if the line should be ignored. For example, the implementation in the base class handles lines like "total 17". On the other hand, if the line should be used for stat'ing, return a false value. """ is_total_line = super().ignores_line(line) my_test = ... return is_total_line or my_test </pre> <p>A <tt class="docutils literal">StatResult</tt> object is similar to the value returned by <a class="reference external" href="https://docs.python.org/library/os.html#os.stat">os.stat</a> and is usually built with statements like</p> <pre class="literal-block"> stat_result = StatResult( (st_mode, st_ino, st_dev, st_nlink, st_uid, st_gid, st_size, st_atime, st_mtime, st_ctime)) stat_result._st_name = ... stat_result._st_target = ... stat_result._st_mtime_precision = ... </pre> <p>with the arguments of the <tt class="docutils literal">StatResult</tt> constructor described in the following table.</p> <table border="1" class="docutils"> <colgroup> <col width="5%" /> <col width="20%" /> <col width="13%" /> <col width="20%" /> <col width="43%" /> </colgroup> <thead valign="bottom"> <tr><th class="head">Index</th> <th class="head">Attribute</th> <th class="head">os.stat type</th> <th class="head"><tt class="docutils literal">StatResult</tt> type</th> <th class="head">Notes</th> </tr> </thead> <tbody valign="top"> <tr><td>0</td> <td>st_mode</td> <td>int</td> <td>int</td> <td> </td> </tr> <tr><td>1</td> <td>st_ino</td> <td>long</td> <td>long</td> <td> </td> </tr> <tr><td>2</td> <td>st_dev</td> <td>long</td> <td>long</td> <td> </td> </tr> <tr><td>3</td> <td>st_nlink</td> <td>int</td> <td>int</td> <td> </td> </tr> <tr><td>4</td> <td>st_uid</td> <td>int</td> <td>str</td> <td>usually only available as string</td> </tr> <tr><td>5</td> <td>st_gid</td> <td>int</td> <td>str</td> <td>usually only available as string</td> </tr> <tr><td>6</td> <td>st_size</td> <td>long</td> <td>long</td> <td> </td> </tr> <tr><td>7</td> <td>st_atime</td> <td>int/float</td> <td>float</td> <td> </td> </tr> <tr><td>8</td> <td>st_mtime</td> <td>int/float</td> <td>float</td> <td> </td> </tr> <tr><td>9</td> <td>st_ctime</td> <td>int/float</td> <td>float</td> <td> </td> </tr> <tr><td>-</td> <td>_st_name</td> <td>-</td> <td>str</td> <td>file name without directory part</td> </tr> <tr><td>-</td> <td>_st_target</td> <td>-</td> <td>str</td> <td>link target (may be absolute or relative)</td> </tr> <tr><td>-</td> <td>_st_mtime_precision</td> <td>-</td> <td>int</td> <td><tt class="docutils literal">st_mtime</tt> precision in seconds</td> </tr> </tbody> </table> <p>If you can't extract all the desirable data from a line (for example, the MS format doesn't contain any information about the owner of a file), set the corresponding values in the <tt class="docutils literal">StatResult</tt> instance to <tt class="docutils literal">None</tt>.</p> <p>Parser classes can use several helper methods which are defined in the class <tt class="docutils literal">Parser</tt>:</p> <ul class="simple"> <li><tt class="docutils literal">parse_unix_mode</tt> parses strings like "drwxr-xr-x" and returns an appropriate <tt class="docutils literal">st_mode</tt> integer value.</li> <li><tt class="docutils literal">parse_unix_time</tt> returns a float number usable for the <tt class="docutils literal"><span class="pre">st_...time</span></tt> values by parsing arguments like "Nov"/"23"/"02:33" or "May"/"26"/"2005". Note that the method expects the timestamp string already split at whitespace.</li> <li><tt class="docutils literal">parse_ms_time</tt> parses arguments like "10-23-01"/"03:25PM" and returns a float number like from <tt class="docutils literal">time.mktime</tt>. Note that the method expects the timestamp string already split at whitespace.</li> </ul> <p>Additionally, there's an attribute <tt class="docutils literal">_month_numbers</tt> which maps lowercase three-letter month abbreviations to integers.</p> <p>For more details, see the two "standard" parsers <tt class="docutils literal">UnixParser</tt> and <tt class="docutils literal">MSParser</tt> in the module <tt class="docutils literal">ftputil/stat.py</tt>.</p> <p>To actually <em>use</em> the parser, call the method <a class="reference internal" href="#set-parser">set_parser</a> of the <tt class="docutils literal">FTPHost</tt> instance.</p> <p>If you can't write a parser or don't want to, please ask on the <a class="reference external" href="https://ftputil.sschwarzer.net/mailinglist">ftputil mailing list</a>. Possibly someone has already written a parser for your server or can help with it.</p> </div> <div class="section" id="faq-tips-and-tricks"> <h1><a class="toc-backref" href="#id30">FAQ / Tips and tricks</a></h1> <div class="section" id="where-can-i-get-the-latest-version"> <h2><a class="toc-backref" href="#id31">Where can I get the latest version?</a></h2> <p>See the <a class="reference external" href="https://ftputil.sschwarzer.net/download">download page</a>. Announcements will be sent to the <a class="reference external" href="https://ftputil.sschwarzer.net/mailinglist">mailing list</a>. Announcements on major updates will also be posted to the <a class="reference external" href="https://mail.python.org/mailman3/lists/python-announce-list.python.org/">Python announcements list</a>.</p> </div> <div class="section" id="is-there-a-mailing-list-on-ftputil"> <h2><a class="toc-backref" href="#id32">Is there a mailing list on <tt class="docutils literal">ftputil</tt>?</a></h2> <p>Yes, please visit <a class="reference external" href="https://ftputil.sschwarzer.net/mailinglist">https://ftputil.sschwarzer.net/mailinglist</a> to subscribe or read the archives.</p> <p>Though you can <em>technically</em> post without subscribing first I can't recommend it: The mails from non-subscribers have to be approved by me and because the arriving mails contain <em>lots</em> of spam, I rarely go through these mails.</p> </div> <div class="section" id="i-found-a-bug-what-now"> <h2><a class="toc-backref" href="#id33">I found a bug! What now?</a></h2> <p>Before reporting a bug, make sure that you already read this manual and tried the <a class="reference external" href="https://ftputil.sschwarzer.net/download">latest version</a> of <tt class="docutils literal">ftputil</tt>. There the bug might have already been fixed.</p> <p>Please see <a class="reference external" href="https://ftputil.sschwarzer.net/issuetrackernotes">https://ftputil.sschwarzer.net/issuetrackernotes</a> for guidelines on entering a bug in <tt class="docutils literal">ftputil</tt>'s ticket system. If you are unsure if the behaviour you found is a bug or not, you should write to the <a class="reference external" href="https://ftputil.sschwarzer.net/mailinglist">ftputil mailing list</a>. <em>Never</em> include confidential information (user id, password, file names, etc.) in the problem report! Be careful!</p> </div> <div class="section" id="does-ftputil-support-tls"> <h2><a class="toc-backref" href="#id34">Does <tt class="docutils literal">ftputil</tt> support TLS?</a></h2> <p><tt class="docutils literal">ftputil</tt> has no <em>built-in</em> TLS support.</p> <p>On the other hand, there are two ways to get TLS support with ftputil:</p> <ul> <li><p class="first">The <tt class="docutils literal">ftplib</tt> library has a class <tt class="docutils literal">FTP_TLS</tt> that you can use for the <tt class="docutils literal">session_factory</tt> keyword argument in the <tt class="docutils literal">FTPHost</tt> constructor. You can't use the class directly though <em>if</em> you need additional setup code in comparison to <tt class="docutils literal">ftplib.FTP</tt>, for example calling <tt class="docutils literal">prot_p</tt>, to secure the data connection. On the other hand, <a class="reference internal" href="#session-factories">ftputil.session.session_factory</a> can be used to create a custom session factory.</p> </li> <li><p class="first">If you have other requirements that <tt class="docutils literal">session_factory</tt> can't fulfill, you may create your own session factory by inheriting from <tt class="docutils literal">ftplib.FTP_TLS</tt>:</p> <pre class="literal-block"> import ftplib import ftputil class FTPTLSSession(ftplib.FTP_TLS): def __init__(self, host, user, password): ftplib.FTP_TLS.__init__(self) self.connect(host, port) self.login(user, password) # Set up encrypted data connection. self.prot_p() ... # Note the `session_factory` parameter. Pass the class, not # an instance. with ftputil.FTPHost(server, user, password, session_factory=FTPTLSSession) as ftp_host: # Use `ftp_host` as usual. ... </pre> </li> </ul> </div> <div class="section" id="how-do-i-connect-to-a-non-default-port"> <h2><a class="toc-backref" href="#id35">How do I connect to a non-default port?</a></h2> <p>By default, an instantiated <tt class="docutils literal">FTPHost</tt> object connects on the usual FTP port. If you have to use a different port, refer to the section <a class="reference internal" href="#session-factories">Session factories</a>.</p> </div> <div class="section" id="how-do-i-set-active-or-passive-mode"> <h2><a class="toc-backref" href="#id36">How do I set active or passive mode?</a></h2> <p>Please see the section <a class="reference internal" href="#session-factories">Session factories</a>.</p> </div> <div class="section" id="how-can-i-debug-an-ftp-connection-problem"> <h2><a class="toc-backref" href="#id37">How can I debug an FTP connection problem?</a></h2> <p>You can do this with a session factory. See <a class="reference internal" href="#session-factories">Session factories</a>.</p> <p>If you want to change the debug level only temporarily after the connection is established, you can reach the <a class="reference internal" href="#session-factories">session object</a> as the <tt class="docutils literal">_session</tt> attribute of the <tt class="docutils literal">FTPHost</tt> instance and call <tt class="docutils literal">_session.set_debuglevel</tt>. Note that the <tt class="docutils literal">_session</tt> attribute should <em>only</em> be accessed for debugging. Calling arbitrary <tt class="docutils literal">ftplib.FTP</tt> methods on the session object may <em>cause</em> bugs!</p> </div> <div class="section" id="conditional-upload-download-to-from-a-server-in-a-different-time-zone"> <h2><a class="toc-backref" href="#id38">Conditional upload/download to/from a server in a different time zone</a></h2> <p>You may find that <tt class="docutils literal">ftputil</tt> uploads or downloads files unnecessarily, or not when it should. Please see the section on <a class="reference internal" href="#time-zone-correction">time zone correction</a>. It may even be sufficient to call <a class="reference internal" href="#synchronize-times">synchronize_times</a>.</p> </div> <div class="section" id="when-i-use-ftputil-all-i-get-is-a-parsererror-exception"> <h2><a class="toc-backref" href="#id39">When I use <tt class="docutils literal">ftputil</tt>, all I get is a <tt class="docutils literal">ParserError</tt> exception</a></h2> <p>The FTP server you connect to may use a directory format that <tt class="docutils literal">ftputil</tt> doesn't understand. You can either write and <a class="reference internal" href="#writing-directory-parsers">plug in your own parser</a> or ask on the <a class="reference external" href="https://ftputil.sschwarzer.net/mailinglist">mailing list</a> for help.</p> </div> <div class="section" id="isdir-isfile-or-islink-incorrectly-return-false"> <h2><a class="toc-backref" href="#id40"><tt class="docutils literal">isdir</tt>, <tt class="docutils literal">isfile</tt> or <tt class="docutils literal">islink</tt> incorrectly return <tt class="docutils literal">False</tt></a></h2> <p>Like Python's counterparts under <a class="reference external" href="https://docs.python.org/library/os.path.html">os.path</a>, <tt class="docutils literal">ftputil</tt>'s methods return <tt class="docutils literal">False</tt> if they can't find the given path.</p> <p>Probably you used <tt class="docutils literal">listdir</tt> on a directory and called <tt class="docutils literal"><span class="pre">is...()</span></tt> on the returned names. But if the argument for <tt class="docutils literal">listdir</tt> wasn't the current directory, the paths won't be found and so all <tt class="docutils literal"><span class="pre">is...()</span></tt> variants will return <tt class="docutils literal">False</tt>.</p> </div> <div class="section" id="i-don-t-find-an-answer-to-my-problem-in-this-document"> <h2><a class="toc-backref" href="#id41">I don't find an answer to my problem in this document</a></h2> <p>Please send an email with your problem report or question to the <a class="reference external" href="https://ftputil.sschwarzer.net/mailinglist">ftputil mailing list</a>, and we'll see what we can do for you. :-)</p> </div> </div> <div class="section" id="bugs-and-limitations"> <h1><a class="toc-backref" href="#id42">Bugs and limitations</a></h1> <ul class="simple"> <li><tt class="docutils literal">ftputil</tt> needs at least Python 3.6 to work.</li> <li>Whether <tt class="docutils literal">ftputil</tt> "sees" "hidden" directory and file names (i. e. names starting with a dot) depends on the configuration of the FTP server. See <a class="reference internal" href="#hidden-files-and-directories">Hidden files and directories</a> for details.</li> <li>Due to the implementation of <tt class="docutils literal">lstat</tt> it can not return a sensible value for the root directory <tt class="docutils literal">/</tt> though stat'ing entries <em>in</em> the root directory isn't a problem. If you know an implementation that can do this, please let me know. The root directory is handled appropriately in <tt class="docutils literal">FTPHost.path.exists/isfile/isdir/islink</tt>, though.</li> <li>In multithreaded programs, you can have each thread use one or more <tt class="docutils literal">FTPHost</tt> instances as long as no instance is shared with other threads.</li> <li>Currently, it is not possible to continue an interrupted upload or download. Contact me if this causes problems for you.</li> <li>There's exactly one cache for <tt class="docutils literal">lstat</tt> results for each <tt class="docutils literal">FTPHost</tt> object, i. e. there's no sharing of cache results determined by several <tt class="docutils literal">FTPHost</tt> objects. See <a class="reference internal" href="#local-caching-of-file-system-information">Local caching of file system information</a> for the reasons.</li> </ul> </div> <div class="section" id="files"> <h1><a class="toc-backref" href="#id43">Files</a></h1> <p>If not overwritten via installation options, the <tt class="docutils literal">ftputil</tt> files reside in the <tt class="docutils literal">ftputil</tt> package. There's also documentation in <a class="reference external" href="https://docutils.sourceforge.net/rst.html">reStructuredText</a> and in HTML format. The locations of these files after installation is system-dependent.</p> <p>The files <tt class="docutils literal"><span class="pre">test_*.py</span></tt> and <tt class="docutils literal">scripted_session.py</tt> are for unit-testing. If you only <em>use</em> <tt class="docutils literal">ftputil</tt>, i. e. <em>don't</em> modify it, you can delete these files.</p> </div> <div class="section" id="references"> <h1><a class="toc-backref" href="#id44">References</a></h1> <ul class="simple"> <li>Postel J, Reynolds J. 1985. <a class="reference external" href="https://www.ietf.org/rfc/rfc959.txt">RFC 959 - File Transfer Protocol (FTP)</a>.</li> <li>Python Software Foundation. 2020. <a class="reference external" href="https://docs.python.org/library/index.html">The Python Standard Library</a>.</li> </ul> </div> <div class="section" id="authors"> <h1><a class="toc-backref" href="#id45">Authors</a></h1> <p><tt class="docutils literal">ftputil</tt> is written by Stefan Schwarzer <<a class="reference external" href="mailto:sschwarzer@sschwarzer.net">sschwarzer@sschwarzer.net</a>> and contributors (see <tt class="docutils literal">doc/contributors.txt</tt>).</p> <p>The original <tt class="docutils literal">lrucache</tt> module was written by Evan Prodromou <<a class="reference external" href="mailto:evan@prodromou.name">evan@prodromou.name</a>>.</p> <p>Feedback is appreciated. :-)</p> </div> </div> </body> </html>