- 6507 Views
- 3 replies
- 3 kudos
Resolved! Is timestamp difference always INTERVAL DAY TO SECOND?
My observations show that timestamp difference has type of INTERVAL DAY TO SECONDS:select typeof(getdate() - current_date()) ----------------------------------------- interval day to secondBut is it guaranteed? Can it be DAY TO MINUTE or, say, YEAR T...
- 6507 Views
- 3 replies
- 3 kudos
- 3 kudos
you can check here for given example: https://docs.databricks.com/sql/language-manual/functions/minussign.htmlthis might help to you.
- 3 kudos
- 5547 Views
- 4 replies
- 3 kudos
Generate sh2 hashkey while loading files to delta table
I have files in azure data lake. I am using autoloader to read the incremental filesfiles don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.In this case i want to ...
- 5547 Views
- 4 replies
- 3 kudos
- 3870 Views
- 3 replies
- 3 kudos
- 3870 Views
- 3 replies
- 3 kudos
- 3 kudos
Df.write.format('jdbc').options( url='jdbc:oracle:thin:@192.168.11.100:1521:ORCL', driver='oracle.jdbc.driver.OracleDriver', dbtable='testschema.test', user='testschema', password='password').mode('overwrite').save()try ...
- 3 kudos
- 12953 Views
- 5 replies
- 6 kudos
split character string in cell with sql
I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...
- 12953 Views
- 5 replies
- 6 kudos
- 6 kudos
you need to use explode functionhttps://stackoverflow.com/questions/61070630/spark-explode-column-with-json-array-to-rows
- 6 kudos
- 3630 Views
- 2 replies
- 3 kudos
How to read enum[] (enum of array) datatype from postgres using spark
We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like belowval jdbcDF3 = spark.read .format("jdbc") .option(...
- 3630 Views
- 2 replies
- 3 kudos
- 3 kudos
You can try custom schema for JDBC read.option("customSchema", "colname STRING")
- 3 kudos
- 2270 Views
- 1 replies
- 1 kudos
Resolved! Apache Spark Programming course -- repo notebooks don't seem to match up with content
I have no idea what I'm doing wrong. I synched this repo <https://github.com/databricks-academy/apache-spark-programming-with-databricks> as instructed. You can see that Notebook 4.1 is query optimization in the 'published' branch.https://github.com/...
- 2270 Views
- 1 replies
- 1 kudos
- 1 kudos
It happened to me also in the past as courses are constantly updated. You can inform the training department through https://help.databricks.com/s/contact-us?ReqType=training
- 1 kudos
- 2588 Views
- 2 replies
- 4 kudos
Resolved! How to Kafka configured on your PC with Databricks?
I'm working on the case to configure Kafka that is installed on my machine (Laptop) & I want to connect it with my Databricks account hosted on the AWS cloud.Secondly, I have CSV files that I want to use for real-time processing from Kafka to Databri...
- 2588 Views
- 2 replies
- 4 kudos
- 4 kudos
For CSV, you need just to readStream in the notebook and append output to CSV using forEachBatch method.Your Kafka on PC needs to have the public address or you need to set AWS VPN and connect from your laptop to be in the same VPC as databricks.
- 4 kudos
- 4343 Views
- 2 replies
- 0 kudos
Databricks SQL Visualizations - export
How can I export Databricks SQL Dashboards as a .pdf?
- 4343 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, I'm trying to export my dashboard to PDF but when I run the export, none of the data is exported, only blank visuals. I would appreciate any help in getting this resolved.Dashboard in DBX: Exported PDF:
- 0 kudos
- 4286 Views
- 3 replies
- 0 kudos
Reading from /tmp no longer working
Since yesterday, reading a file copied into the cluster is no longer working.What used to work:blob = gcs_bucket.get_blob("dev/data.ndjson") -> worksblob.download_to_filename("/tmp/data-copy.ndjson") -> worksdf = spark.read.json("/tmp/data-copy.ndjso...
- 4286 Views
- 3 replies
- 0 kudos
- 0 kudos
I encountered this same issue, and figured out a fix!For some reason, it seems like only %sh cells can access the /tmp directory. So I just did...%sh ch /tmp/<file> /dbfs/<desired-location> and then accessed it form there using Spark.
- 0 kudos
- 1794 Views
- 3 replies
- 2 kudos
I haven't received certificate or badge after clearing data engineer associate exam
Hello Databricks Team,I completed my data engineer associate exam & passed the exam. But still now I haven't received certificate or badge. Also created help ticket, Ticket Number: #[00232273]​ Mail id : satishshravan888@gmail.comThanks,Satish
- 1794 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Satish Shravan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 2 kudos
- 14684 Views
- 3 replies
- 6 kudos
Resolved! "User not authorized" error when trying to deploy access connector on Azure
Hi,I have been trying to deploy Access Connector resource on Azure using Azure Pipelines (YAML) and a Bicep template but I cannot find a solution to this error:ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resou...
- 14684 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi,I fixed this issue by adding the service principal to the list of service principals in the Account Console. My guess is that after the access connector is created an API call is made to the Databricks account and the service principal making that...
- 6 kudos
- 3627 Views
- 2 replies
- 2 kudos
implement autoloader to ingest data into delta lake, i have 100 different tables with full load , append merge senarios
i want to implement autoloader to ingest data into delta lake from 5 different source systems and i have 100 different tables in each database how do we dynamically address this by using autoloader , trigger once option - full load , append merge sen...
- 3627 Views
- 2 replies
- 2 kudos
- 2 kudos
You can create a generic notebook that will be parametrized with the table name/source system and then just simply trigger notebook with different parameters (for each table/source system).For parametrization you can use dbutils.widgets (https://docs...
- 2 kudos
- 5103 Views
- 2 replies
- 4 kudos
Why does Databricks SQL drop ending 0 in decimal data type
All,I have a column, RateAdj that is defined as DECIMAL(15,5) and I can see that the value is 4.00000, but when this gets inserted into my table it shows as just 4.%sql SELECT LTRIM(RTRIM(IFNULL(FORMAT_NUMBER(RateADJ, '0.00000'), '0.00000')))This i...
- 5103 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Greg Bradley​ Hope everything is going great.Checking in. If @Uma Maheswara Rao Desula​ answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?Thanks!
- 4 kudos
- 2767 Views
- 1 replies
- 0 kudos
Resolved! Error Using spark.catalog.dropTempView()
I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a not...
- 2767 Views
- 1 replies
- 0 kudos
- 0 kudos
spark.sql("DROP TABLE "+prefix_updates) spark.sql("DROP TABLE "+prefix_main)Fixed it for me.
- 0 kudos
- 3237 Views
- 4 replies
- 4 kudos
Passed Data Engineer Associate Exam but received twice the Lakehouse Accreditation
Hi all,today (27/12/22 14:00 Rome Time Zone) I passed the Data Engineer Associate exam, but I received the badge of Lakehouse Fundamentals (second time). My email address is: sisbarra@gmail.comMy company address is: antonio.sisbarra@nttdata.comCan ...
- 3237 Views
- 4 replies
- 4 kudos
- 4 kudos
Hello Antonio,I deleted the badge under: antonio.sisbarra@nttdata.com . You are good to go.Thanks!
- 4 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
