- 21124 Views
- 2 replies
- 12 kudos
Resolved! How can Delta table protocol version be downgraded from higher version to lower version the table properties minReader from 2 to 1 and MaxWriter from 5 to 3.
Is there a possibility to downgrade the Delta Table protocol versions minReader from 2 to 1 and maxWriter from 5 to 3? I have set the TBL properties to 2 and 5 and columnmapping mode to rename the columns in the DeltaTable but the other users are rea...
- 21124 Views
- 2 replies
- 12 kudos
- 12 kudos
Unfortunately You can't downgrade the version. it's an irreversible operation.
- 12 kudos
- 1592 Views
- 0 replies
- 2 kudos
get diagnostics using pyspark instead of microsoft diagnostics
I know you can use microsoft diagnostics to receive notifications such as commandSubmit or commandFinish in json. But is it possible to get these diagnostics in databricks itself using pyspark?I would like to get these diagnostics inside of a noteboo...
- 1592 Views
- 0 replies
- 2 kudos
- 13084 Views
- 3 replies
- 3 kudos
Resolved! Databricks SQL dashboard refresh
We have scheduled a dashboard to automatically refresh at some specific time.However, some visualizations in the dashboard don't get refreshed at the scheduled time.Checking the query logs, we found the source queries were properly executed, but the ...
- 13084 Views
- 3 replies
- 3 kudos
- 3 kudos
Thank you for the answers.I'm using Databricks SQL environment, not the Data Science & Engineering one. And, I scheduled the dashboard following this guideline: https://docs.databricks.com/sql/user/dashboards/index.html#automatically-refresh-a-dashbo...
- 3 kudos
- 3167 Views
- 3 replies
- 2 kudos
Which is quicker: grouping a table that is a join of several others or querying data?
This may be a tricky question, so please bear with meIn a real life scenario, i have a dataframe (i'm using pyspark) called age, with is a groupBy of other 4 dataframes. I join these 4 so at the end I have a few million rows, but after the groupBy th...
- 3167 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Marcos Dias​ ,Frankly, I think we need more detail to answer your question:Are these 4 dataframes​ updated their data?How often you use the groupBy-dataframe?
- 2 kudos
- 6610 Views
- 5 replies
- 4 kudos
remote database connection error
Hi,I am using databricks through azure. I am trying to connect to remote oracle database using jdbc url. I am getting an error of no suitable driver found."java.sql.SQLException: No suitable driver"can somebody help mw with this?
- 6610 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @vikas k​ Hope all is well! Does @Hubert Dudek​ response were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 2053 Views
- 0 replies
- 2 kudos
Date Formats while extracting data from Cosmos Mongo DB using Azure Databricks.
I have been trying to extract one date field from cosmos which looks like as below:"lastModifiedDate" : { "$date" : 1668443121840 }when the above field is extracted using Databricks it gets converted into a date format which looks like this...
- 2053 Views
- 0 replies
- 2 kudos
- 4707 Views
- 2 replies
- 1 kudos
Resolved! Use API to Clone Dashboards with Widgets in Databricks SQL?
Hi, I manually created a template dashboard with one widget. I wanted to clone this dashboard using the Create Dashboard API however I don't know what to put for the widget object. What I did was use the Retrieve API on the template dashboard, and th...
- 4707 Views
- 2 replies
- 1 kudos
- 1 kudos
@Akash Bhat​ how do I clone a dashboard across workspaces? Being able to deploy dashboards (with widgets!) through the API is essential to set up proper Data Engineering workflows.
- 1 kudos
- 1707 Views
- 0 replies
- 2 kudos
In person Databricks meetup in Singapore?
Hi I would like to know if anyone interested to volunteer in person Databricks meetup.Please share your thoughts, and we can talk further about the logistics Thank you
- 1707 Views
- 0 replies
- 2 kudos
- 1181 Views
- 0 replies
- 0 kudos
Simple PySpark query very slow in pushing to snowflake
Hi all,​I have a question which is likely very fundamental. I am passing data from hive to snowflake using pyspark. My query is very simple - "select from table limit 100".​ The table I am querying is very large, but this query can be shown to the co...
- 1181 Views
- 0 replies
- 0 kudos
- 5858 Views
- 2 replies
- 2 kudos
Git Integration Configuration via Command Line or API
I have an Azure service principle that is used for our CI/CD pipelines. We do not have access to the Databricks UI via user logins. Our github repos also require SSO PATs. How can I configure the git integration for the service principal so that I ca...
- 5858 Views
- 2 replies
- 2 kudos
- 4217 Views
- 0 replies
- 1 kudos
How to restore a dbfs directory ?
Hello,Can I restore a dbfs directory ?I dropped a database using an important directory Someone can help me please ?
- 4217 Views
- 0 replies
- 1 kudos
- 61311 Views
- 25 replies
- 37 kudos
Fatal error: Python kernel is unresponsive
Hey guys, I'm using petastorm to train DNN, First i convert spark df with make_spark_convertor and then open a reader on the materialized dataset.While i start training session only on subset of the data every thing works fine but when I'm using all...
- 61311 Views
- 25 replies
- 37 kudos
- 37 kudos
Same error. This started a few days ago on notebooks that used to run fine in the past. Now, I cannot finish a notebook.I have already disabled almost all output being streamed to the result buffer, but the problem persists. I am left with <50 lines ...
- 37 kudos
- 3424 Views
- 3 replies
- 3 kudos
log4j vulnerability - action plan for clients
I'm looking for some information regarding log4j vulnerability - if any databricks runtime should be changed manually by the client or when specific update will be applied.I know I can go through the docs by myself, finding out which log4j library is...
- 3424 Views
- 3 replies
- 3 kudos
- 3 kudos
Thanks for sharing such important facts. I got the best security service provider website on the internet. They are excellence in the field of security.
- 3 kudos
- 15227 Views
- 7 replies
- 9 kudos
Incorrectly truncated long numbers in DS&E workspaces notebooks
Hello,I am getting inconsistent representation of long types.1661817599972 is the unix timestamp in milliseconds for Monday, August 29, 2022 11:59:59.972 PM GMTwhen I execute:`select 1661817599972 as t`the result is:166181759997 (last digit truncated...
- 15227 Views
- 7 replies
- 9 kudos
- 9 kudos
Hi @Cosimo Felline​ Hope all is well!Does @Hubert Dudek (Customer)​ response answer your question?If you could resolve your issue, would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd...
- 9 kudos
- 16477 Views
- 11 replies
- 10 kudos
Error in SQL statement: AnalysisException: Cannot up cast documents from array
Hi Everyone,I am getting the following error when running a SQL query and do not understand what it means or what can be done to resolve it. Any recommendations?View DDL:CREATE VIEW myschema.table ( accountId, agreementType, capture_file_name, ...
- 16477 Views
- 11 replies
- 10 kudos
- 10 kudos
Hi @Michael Okulik​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 10 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
