Xibo

Support files greater than 2.2Gb in size of 32 bit systems

Bug #696962 reported by Matthew Holder
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Xibo
Confirmed
Medium
Xibo Developers

Bug Description

The offset of a file being uploaded or downloaded is defined by an integer, on 32 bit systems this is represented by 32 bits.

This number is approximately 2.2Gb.

I have tested this using the API, uploading to a 1.3 (type) release and when I hit the limit the offset returned by the server is the negative limit of a 32bit integer.

The database stores filesize as a BIGINT, so this will not need to be changed

It looks to me that this bug will affect:

1) The way the server reports offsets
2) The way clients and any other apps deal with offsets

I can see two ways of proceeding

1) Change the server reporting to the long data type and then change how the clients deal
2) Change the server reporting to sending two offset values, offset and offset_high. The clients and applications can then put together the two numbers with bit shifting of the offset_high to create the long value for the offset.

Option two would mean that clients can remain backwards compatible for files less than 2.2Gb.

I am not sure how much of a consideration backwards compatibility is, especially if offsets are reported as long from a new major revision.

Matt

Related branches

lp://staging/~alexharrington/xibo/pyclient-1.2.1a1 (Merged)
lp://staging/~dangarner/xibo/client-170-alpha2
Xibo Maintainters: Pending requested
Diff: 1291 lines (+337/-382)
20 files modified
client/dotNET/Forms/OptionForm.cs (+1/-1)
client/dotNET/Log/ClientInfo.Designer.cs (+33/-27)
client/dotNET/Log/ClientInfo.cs (+8/-2)
client/dotNET/Log/ClientInfo.resx (+21/-18)
client/dotNET/Log/StatLog.cs (+63/-125)
client/dotNET/Logic/ApplicationSettings.cs (+2/-2)
client/dotNET/Logic/RequiredFiles.cs (+3/-3)
client/dotNET/Logic/ScheduleManager.cs (+41/-33)
client/dotNET/Logic/ScreenShot.cs (+1/-0)
client/dotNET/Media/CefWebMedia.cs (+69/-40)
client/dotNET/Media/IeWebMedia.cs (+70/-41)
client/dotNET/Media/ShellCommand.cs (+5/-1)
client/dotNET/Media/TemporaryFile.cs (+0/-71)
client/dotNET/Web References/xmds/Reference.cs (+10/-8)
client/dotNET/Web References/xmds/Reference.map (+1/-1)
client/dotNET/Web References/xmds/xmds.wsdl (+4/-3)
client/dotNET/XiboClient.csproj (+2/-3)
client/dotNET/XmdsAgents/FileAgent.cs (+1/-1)
client/dotNET/XmdsAgents/RegisterAgent.cs (+1/-1)
client/dotNET/default.config.xml (+1/-1)
lp://staging/~dangarner/xibo/server-170-alpha2
Xibo Maintainters: Pending requested
Diff: 30604 lines (+23044/-3722)
174 files modified
example_oauth/index.php (+14/-11)
server/config/client.config.php (+1/-1)
server/config/config.class.php (+6/-4)
server/install/database/80.sql (+1/-6)
server/install/database/81.sql (+23/-0)
server/install/master/data.sql (+6/-7)
server/install/master/structure.sql (+5/-3)
server/lib/app/cache.class.php (+82/-0)
server/lib/app/debug.class.php (+180/-169)
server/lib/app/kit.class.php (+456/-445)
server/lib/app/modulemanager.class.php (+2/-5)
server/lib/app/permissionmanager.class.php (+1/-3)
server/lib/app/responsemanager.class.php (+3/-1)
server/lib/app/session.class.php (+11/-4)
server/lib/app/thememanager.class.php (+56/-0)
server/lib/app/translationengine.class.php (+69/-68)
server/lib/data/campaign.data.class.php (+3/-1)
server/lib/data/campaignsecurity.data.class.php (+9/-0)
server/lib/data/datasetdata.data.class.php (+4/-0)
server/lib/data/datasetgroupsecurity.data.class.php (+18/-40)
server/lib/data/display.data.class.php (+133/-2)
server/lib/data/displayprofile.data.class.php (+55/-0)
server/lib/data/layout.data.class.php (+10/-18)
server/lib/data/maintenance.data.class.php (+156/-15)
server/lib/data/media.data.class.php (+101/-5)
server/lib/data/region.data.class.php (+5/-10)
server/lib/data/schedule.data.class.php (+578/-516)
server/lib/data/stat.data.class.php (+121/-63)
server/lib/data/usergroup.data.class.php (+52/-0)
server/lib/include.php (+8/-2)
server/lib/modules/module.class.php (+53/-47)
server/lib/pages/admin.class.php (+37/-71)
server/lib/pages/campaign.class.php (+36/-29)
server/lib/pages/content.class.php (+6/-5)
server/lib/pages/display.class.php (+53/-73)
server/lib/pages/displaygroup.class.php (+15/-25)
server/lib/pages/index.class.php (+10/-0)
server/lib/pages/install.class.php (+7/-1)
server/lib/pages/layout.class.php (+102/-59)
server/lib/pages/log.class.php (+4/-2)
server/lib/pages/module.class.php (+5/-4)
server/lib/pages/oauth.class.php (+1/-0)
server/lib/pages/preview.class.php (+3/-1)
server/lib/pages/schedule.class.php (+13/-12)
server/lib/pages/stats.class.php (+153/-9)
server/lib/pages/template.class.php (+14/-25)
server/lib/pages/timeline.class.php (+239/-57)
server/lib/pages/user.class.php (+4/-3)
server/lib/service/rest.class.php (+15/-2)
server/lib/service/service.wsdl (+3/-2)
server/lib/service/xmdssoap.class.php (+120/-90)
server/lib/xmds.inc.php (+8/-5)
server/locale/dbtranslate.php (+4/-0)
server/maintenance.php (+119/-172)
server/manual/content/admin/release_notes_1.7.0-alpha2.php (+84/-0)
server/manual/content/routes.php (+1/-0)
server/manual/content/toc_developer_releasenotes.php (+1/-0)
server/manual/template.php (+1/-1)
server/modules/3rdparty/forecast.php (+38/-0)
server/modules/clock.module.php (+33/-48)
server/modules/datasetview.module.php (+41/-14)
server/modules/embedded.module.php (+247/-193)
server/modules/font.module.php (+187/-0)
server/modules/forecastio.module.php (+508/-0)
server/modules/module_user_general.php (+39/-14)
server/modules/preview/Html4TransitionalTemplate.html (+26/-0)
server/modules/preview/HtmlTemplate.html (+9/-3)
server/modules/preview/HtmlTemplateForGetResource.html (+0/-155)
server/modules/preview/HtmlTemplateSimple.html (+1/-1)
server/modules/preview/fonts.css (+11/-0)
server/modules/preview/html-preview.js (+1/-1)
server/modules/preview/vendor/jquery-cycle-2.1.6.min.js (+16/-0)
server/modules/preview/vendor/jquery.marquee.min.js (+5/-0)
server/modules/preview/xibo-dataset-render.js (+51/-0)
server/modules/preview/xibo-layout-scaler.js (+62/-0)
server/modules/preview/xibo-text-render.js (+43/-211)
server/modules/preview/xibo-webpage-render.js (+81/-69)
server/modules/text.module.php (+150/-102)
server/modules/theme/HtmlTemplateForClock.html (+1/-1)
server/modules/theme/HtmlTemplateForFlipClock.html (+1/-1)
server/modules/theme/forecastio/weather_icons/weather-icons.min.css (+23/-0)
server/modules/theme/forecastio/weather_icons/weathericons-regular-webfont.svg (+153/-0)
server/modules/ticker.module.php (+93/-39)
server/modules/webpage.module.php (+303/-189)
server/services.php (+4/-2)
server/theme/default/css/html-preview.css (+2/-2)
server/theme/default/html/campaign_form_layout_assign.php (+2/-19)
server/theme/default/html/campaign_form_layout_assign_list.php (+0/-39)
server/theme/default/html/footer.php (+27/-7)
server/theme/default/html/form_grid_pager.php (+0/-41)
server/theme/default/html/grid_pager.php (+14/-15)
server/theme/default/html/grid_render.php (+1/-0)
server/theme/default/html/header.php (+7/-6)
server/theme/default/html/layout_designer.php (+2/-5)
server/theme/default/html/schedule_page.php (+1/-1)
server/theme/default/html/settings_page.php (+103/-86)
server/theme/default/html/stats_page_availability.php (+47/-0)
server/theme/default/html/stats_page_grid.php (+0/-6)
server/theme/default/html/status_dashboard.php (+2/-2)
server/theme/default/html/table_render.php (+110/-48)
server/theme/default/js/xibo-calendar.js (+8/-7)
server/theme/default/js/xibo-cms.js (+347/-63)
server/theme/default/js/xibo-forms.js (+24/-10)
server/theme/default/libraries/bootstrap-ekko-lightbox/ekko-lightbox.min.css (+6/-0)
server/theme/default/libraries/bootstrap-ekko-lightbox/ekko-lightbox.min.js (+7/-0)
server/theme/default/libraries/calendar/js/language/de-DE.js (+0/-73)
server/theme/default/libraries/calendar/js/language/de.js (+73/-0)
server/theme/default/libraries/ckeditor/config.js (+4/-1)
server/theme/default/libraries/ckeditor/contents.css (+2/-2)
server/theme/default/libraries/jquery-message-queuing/jquery.ba-jqmq.min.js (+9/-0)
server/theme/default/libraries/jquery-tablesorter/addons/pager/jquery.tablesorter.pager.css (+42/-0)
server/theme/default/libraries/jquery-tablesorter/addons/pager/jquery.tablesorter.pager.js (+958/-0)
server/theme/default/libraries/jquery-tablesorter/addons/pager/jquery.tablesorter.pager.min.js (+2/-0)
server/theme/default/libraries/jquery-tablesorter/css/bootstrap.less (+316/-0)
server/theme/default/libraries/jquery-tablesorter/css/filter.formatter.css (+183/-0)
server/theme/default/libraries/jquery-tablesorter/css/metro.less (+351/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.black-ice.css (+186/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.blue.css (+221/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.bootstrap.css (+152/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.bootstrap_2.css (+150/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.dark.css (+187/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.default.css (+189/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.dropbox.css (+212/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.green.css (+203/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.grey.css (+245/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.ice.css (+201/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.jui.css (+156/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.less (+323/-0)
server/theme/default/libraries/jquery-tablesorter/css/theme.metro-dark.css (+192/-0)
server/theme/default/libraries/jquery-tablesorter/js/extras/jquery.quicksearch.js (+191/-0)
server/theme/default/libraries/jquery-tablesorter/js/extras/semver-mod.js (+1026/-0)
server/theme/default/libraries/jquery-tablesorter/js/extras/semver.js (+1011/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.metadata.js (+116/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.tablesorter.js (+1901/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.tablesorter.min.js (+5/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.tablesorter.widgets-filter-formatter-select2.js (+138/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.tablesorter.widgets-filter-formatter.js (+1145/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.tablesorter.widgets-filter-formatter.min.js (+6/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.tablesorter.widgets.js (+1912/-0)
server/theme/default/libraries/jquery-tablesorter/js/jquery.tablesorter.widgets.min.js (+17/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-date-extract.js (+81/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-date-iso8601.js (+34/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-date-month.js (+33/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-date-two-digit-year.js (+74/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-date-weekday.js (+33/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-date.js (+36/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-duration.js (+40/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-feet-inch-fraction.js (+63/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-file-type.js (+73/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-ignore-articles.js (+61/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-image.js (+20/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-input-select.js (+161/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-ipv6.js (+76/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-metric.js (+77/-0)
server/theme/default/libraries/jquery-tablesorter/js/parsers/parser-roman.js (+117/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-alignChar.js (+145/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-build-table.js (+453/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-columnSelector.js (+317/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-cssStickyHeaders.js (+70/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-editable.js (+200/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-grouping.js (+249/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-headerTitles.js (+91/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-math.js (+413/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-output.js (+316/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-pager.js (+975/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-print.js (+123/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-reflow.js (+179/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-repeatheaders.js (+50/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-scroller.js (+218/-0)
server/theme/default/libraries/jquery-tablesorter/js/widgets/widget-staticRow.js (+124/-0)
server/theme/default/libraries/jquery/jquery.metadata.js (+0/-116)
server/theme/default/libraries/jquery/jquery.tablesorter.pack.js (+0/-6)
server/theme/default/libraries/jquery/jquery.tablesorter.pager.css (+0/-28)
server/theme/default/libraries/jquery/jquery.tablesorter.pager.js (+0/-2)
Matthew Holder (matt-mattmole)
Changed in xibo:
status: New → Opinion
Revision history for this message
Alex Harrington (alexharrington) wrote :

Python Client / Offline Download Client Implications:
 * Need to ensure offsets are converted to long types
 * Need to ensure all places dealing with file size are converted to long types

API Testsuite Implications:
 * Need to ensure offsets are converted to long types
 * Need to ensure that filesizes are represented as longs

Revision history for this message
Dan Garner (dangarner) wrote :

Server / API implications
 * None as PHP will treat larger than INT values as a FLOAT automatically

.NET client implications
 * Will need to change the ints to doubles in the FileCollector

Dan Garner (dangarner)
no longer affects: xibo/1.3
Changed in xibo:
milestone: 1.4.0-rc1 → 1.5.0
Alex Harrington (alexharrington)
Changed in xibo:
milestone: 1.5.0 → 1.7.0
Dan Garner (dangarner)
Changed in xibo:
milestone: 1.7.0 → 1.7.0-alpha
Dan Garner (dangarner)
Changed in xibo:
milestone: 1.7.0-alpha → 1.7.0-alpha2
Revision history for this message
Dan Garner (dangarner) wrote :

I've untagged this because it is fundamentally not supported in PHP at the moment - the majority of file based functions will return incorrect data for files > 2gb.

Changed in xibo:
milestone: 1.7.0-alpha2 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.