- 4224 Views
- 1 replies
- 2 kudos
Resolved! org.apache.http.conn.ConnectTimeoutException: What does this mean and how can we resolve it.
My team has run into getting this error pretty frequently on one of our larger jobs. I've set out retry policy to 5 and that seems to fix it and keep the job going. It seems like it's unable to pick up the task immediately but can after it's complete...
- 4224 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey @Coleman Milligan​ ,I also faced this type of issue many times you can add the below configuration in your cluster and it should work.spark.executor.heartbeatInterval 60sspark.network.timeout 120sFor more details, you can explore this doc - https...
- 2 kudos
- 5808 Views
- 2 replies
- 2 kudos
cannot convert Parquet type INT64 to Photon type double
I am trying to read in files via the COPY INTO command but I am getting this error lately for a certain subset of the data;`Error while reading file: Schema conversion error: cannot convert Parquet type INT64 to Photon type double`These are my option...
- 5808 Views
- 2 replies
- 2 kudos
- 2 kudos
hey @Andrew Fogarty​ I also faced the same issue when I moved from the 7.3 LTS version to a higher runtime version so to mitigate this issue you can use the below cluster configuration spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsS...
- 2 kudos
- 3316 Views
- 4 replies
- 0 kudos
Resolved! Safari problems after the maintenance on 12/9/2022
I'm experience some problems on Safari 15.3 ( MacOS )I would like to know if I am alone in this and how to fix ( if I can ) this.This is the Databricks SQLData science and Engineering. ( is this case Workflows).
- 3316 Views
- 4 replies
- 0 kudos
- 3853 Views
- 0 replies
- 0 kudos
i want to plot multiple data frames from a pandas datafreame
Hi all, i want to plot multiple charts from a pandas datafreame. However, when i run the code below it says "Command result size exceeds limit: Exceeded 20971520 bytes (current = 20973124)". If I move line 11 and place at 21 (outside of the functi...
- 3853 Views
- 0 replies
- 0 kudos
- 4918 Views
- 1 replies
- 1 kudos
Jobs failing with repl error
Recently my Databricks jobs have failed with the error message:Failure starting repl. Try detaching and re-attaching the notebook. java.lang.Exception: Python repl did not start in 30 seconds seconds. at com.databricks.backend.daemon.driver.Ipyker...
- 4918 Views
- 1 replies
- 1 kudos
- 1 kudos
Yes, you can use re-try if still it's not resolve raise a support ticket to databricks
- 1 kudos
- 22207 Views
- 2 replies
- 2 kudos
Can I query my Delta tables with PowerBI?
I would like to connect to the Delta tables I have created with PowerBI to use for reporting. Is it possible to do this with Databricks or do I have to write my data to some other serving layer?
- 22207 Views
- 2 replies
- 2 kudos
- 2 kudos
if you want to read your Delta Lake table directly from the storage without the need of having a Databricks cluster up and running you can also use the official connector Power BI connector for Delta Lake https://github.com/delta-io/connectors/tree/m...
- 2 kudos
- 3158 Views
- 4 replies
- 11 kudos
Resolved! Delta table output to Databricks SQL
How to write a delta table output to Databricks SQL for analysis purpose.
- 3158 Views
- 4 replies
- 11 kudos
- 1772 Views
- 1 replies
- 5 kudos
Resolved! Trigger another .py file by uisng 2 .py files.
Hi,I have 3 .py files - a.py, b.py & c.py files. By joining a.py & b.py, based on the output that I get need to trigger the c.py file.
- 1772 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi @KVNARK .​ refer below link this will help in thisLink
- 5 kudos
- 6216 Views
- 2 replies
- 6 kudos
Is there a function similar to split_part, json_extract_scalar?
I am using spark_sql version 3.2.1. Is there a function that can replacesplit_part,json_extract_scalarare not?
- 6216 Views
- 2 replies
- 6 kudos
- 6 kudos
pyspark.sql.functions.get_json_object(col, path)[source]Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.​
- 6 kudos
- 1548 Views
- 1 replies
- 0 kudos
Hi Nadia, Avail Free Exam Vouchers
Hi Nadia,I am preparing for multiple databricks certifications. Could you please provide me any events links to my email address "databrickscertificates.2022.23@gmail.com" so that I can register to the event and avail any FREE vouchers for exams.
- 1548 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Rey, There are currently no events running that are offering free vouchers. We are offering 75% vouchers. Please check out our events page for future events: https://www.databricks.com/learn/training/homeThank you!
- 0 kudos
- 3194 Views
- 1 replies
- 0 kudos
AutoLoader - process multiple files
I need to process files of different schema coming to different folders in ADLS using Autoloader. Do I need to start a separate read stream for each file type / folder or can this be handled using a single stream ?When I tried using a single stream, ...
- 3194 Views
- 1 replies
- 0 kudos
- 0 kudos
As you are talking about different schemas ,perhaps schemaevolutionmode, infercolumntypes, or schemahints may help?? Check out this- 32min onward - https://youtu.be/8a38Fv9cpd8 ​Hope it helps, do let know how you solve it if you can.​
- 0 kudos
- 3966 Views
- 2 replies
- 1 kudos
Resolved! Cancelling the exam- need to know whats policy if had scheduled the exam with voucher
I have my exam scheduled for next month ,but I am going to cancel it( i have regestered this exam using a voucher, In future i may schedule other exam ,would i be able to utilize that voucher that i used for the exam am gonna cancel? I mean could tha...
- 3966 Views
- 2 replies
- 1 kudos
- 1 kudos
No, once redeemed means you cannot use the voucher again, better reschedule the exam now itself.
- 1 kudos
- 1989 Views
- 1 replies
- 4 kudos
Documentation Update  Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data enginee...
Documentation Update Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, and Databricks ...
- 1989 Views
- 1 replies
- 4 kudos
- 4 kudos
Thanks for sharing @Sujitha Ramamoorthy​
- 4 kudos
- 2437 Views
- 1 replies
- 4 kudos
Resolved! Databricks SQL Connector Abstraction for Python
Databricks SQL framework, easy to learn, fast to code, ready for production.I built an abstraction of the databricks-sql-connector in order to follow a pattern closer to the concepts of ORM tools, in addition to facilitating the adoption of the data ...
- 2437 Views
- 1 replies
- 4 kudos
- 4 kudos
Sure, will try and provide feedback for same
- 4 kudos
- 4188 Views
- 1 replies
- 2 kudos
Resolved! Identity column in DLT using Python
How would I implement the Identity column in Delta Live Tables using Python syntax?GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( [ START WITH start ] [ INCREMENT BY step ] ) ]
- 4188 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Kory Skistad​ Please find below the table schema definition to use in a python dlt pipeline. You can see it mentions the identity column definition. @dlt.table( comment="Raw data on sales", schema=""" customer_id STRING, customer_name STR...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
