- 10286 Views
- 9 replies
- 7 kudos
Calling a python function (def) in databricks
Not sure if I'm missing something here, but running a task outside of a python function runs much much quicker than executing the same task inside a function. Is there something I'm missing with how spark handles functions? 1) def task(x): y = dostuf...
- 10286 Views
- 9 replies
- 7 kudos
- 7 kudos
don't use python normal function use UDF in pyspark so that will be faster
- 7 kudos
- 7575 Views
- 3 replies
- 3 kudos
Resolved! Is timestamp difference always INTERVAL DAY TO SECOND?
My observations show that timestamp difference has type of INTERVAL DAY TO SECONDS:select typeof(getdate() - current_date()) ----------------------------------------- interval day to secondBut is it guaranteed? Can it be DAY TO MINUTE or, say, YEAR T...
- 7575 Views
- 3 replies
- 3 kudos
- 3 kudos
you can check here for given example: https://docs.databricks.com/sql/language-manual/functions/minussign.htmlthis might help to you.
- 3 kudos
- 6324 Views
- 4 replies
- 3 kudos
Generate sh2 hashkey while loading files to delta table
I have files in azure data lake. I am using autoloader to read the incremental filesfiles don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.In this case i want to ...
- 6324 Views
- 4 replies
- 3 kudos
- 4374 Views
- 3 replies
- 3 kudos
- 4374 Views
- 3 replies
- 3 kudos
- 3 kudos
Df.write.format('jdbc').options( url='jdbc:oracle:thin:@192.168.11.100:1521:ORCL', driver='oracle.jdbc.driver.OracleDriver', dbtable='testschema.test', user='testschema', password='password').mode('overwrite').save()try ...
- 3 kudos
- 13927 Views
- 5 replies
- 6 kudos
split character string in cell with sql
I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...
- 13927 Views
- 5 replies
- 6 kudos
- 6 kudos
you need to use explode functionhttps://stackoverflow.com/questions/61070630/spark-explode-column-with-json-array-to-rows
- 6 kudos
- 4072 Views
- 2 replies
- 3 kudos
How to read enum[] (enum of array) datatype from postgres using spark
We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like belowval jdbcDF3 = spark.read .format("jdbc") .option(...
- 4072 Views
- 2 replies
- 3 kudos
- 3 kudos
You can try custom schema for JDBC read.option("customSchema", "colname STRING")
- 3 kudos
- 2591 Views
- 1 replies
- 1 kudos
Resolved! Apache Spark Programming course -- repo notebooks don't seem to match up with content
I have no idea what I'm doing wrong. I synched this repo <https://github.com/databricks-academy/apache-spark-programming-with-databricks> as instructed. You can see that Notebook 4.1 is query optimization in the 'published' branch.https://github.com/...
- 2591 Views
- 1 replies
- 1 kudos
- 1 kudos
It happened to me also in the past as courses are constantly updated. You can inform the training department through https://help.databricks.com/s/contact-us?ReqType=training
- 1 kudos
- 2929 Views
- 2 replies
- 4 kudos
Resolved! How to Kafka configured on your PC with Databricks?
I'm working on the case to configure Kafka that is installed on my machine (Laptop) & I want to connect it with my Databricks account hosted on the AWS cloud.Secondly, I have CSV files that I want to use for real-time processing from Kafka to Databri...
- 2929 Views
- 2 replies
- 4 kudos
- 4 kudos
For CSV, you need just to readStream in the notebook and append output to CSV using forEachBatch method.Your Kafka on PC needs to have the public address or you need to set AWS VPN and connect from your laptop to be in the same VPC as databricks.
- 4 kudos
- 5758 Views
- 2 replies
- 0 kudos
Databricks SQL Visualizations - export
How can I export Databricks SQL Dashboards as a .pdf?
- 5758 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, I'm trying to export my dashboard to PDF but when I run the export, none of the data is exported, only blank visuals. I would appreciate any help in getting this resolved.Dashboard in DBX: Exported PDF:
- 0 kudos
- 4717 Views
- 3 replies
- 0 kudos
Reading from /tmp no longer working
Since yesterday, reading a file copied into the cluster is no longer working.What used to work:blob = gcs_bucket.get_blob("dev/data.ndjson") -> worksblob.download_to_filename("/tmp/data-copy.ndjson") -> worksdf = spark.read.json("/tmp/data-copy.ndjso...
- 4717 Views
- 3 replies
- 0 kudos
- 0 kudos
I encountered this same issue, and figured out a fix!For some reason, it seems like only %sh cells can access the /tmp directory. So I just did...%sh ch /tmp/<file> /dbfs/<desired-location> and then accessed it form there using Spark.
- 0 kudos
- 2041 Views
- 3 replies
- 2 kudos
I haven't received certificate or badge after clearing data engineer associate exam
Hello Databricks Team,I completed my data engineer associate exam & passed the exam. But still now I haven't received certificate or badge. Also created help ticket, Ticket Number: #[00232273]​ Mail id : satishshravan888@gmail.comThanks,Satish
- 2041 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Satish Shravan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 2 kudos
- 15841 Views
- 3 replies
- 6 kudos
Resolved! "User not authorized" error when trying to deploy access connector on Azure
Hi,I have been trying to deploy Access Connector resource on Azure using Azure Pipelines (YAML) and a Bicep template but I cannot find a solution to this error:ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resou...
- 15841 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi,I fixed this issue by adding the service principal to the list of service principals in the Account Console. My guess is that after the access connector is created an API call is made to the Databricks account and the service principal making that...
- 6 kudos
- 4062 Views
- 2 replies
- 2 kudos
implement autoloader to ingest data into delta lake, i have 100 different tables with full load , append merge senarios
i want to implement autoloader to ingest data into delta lake from 5 different source systems and i have 100 different tables in each database how do we dynamically address this by using autoloader , trigger once option - full load , append merge sen...
- 4062 Views
- 2 replies
- 2 kudos
- 2 kudos
You can create a generic notebook that will be parametrized with the table name/source system and then just simply trigger notebook with different parameters (for each table/source system).For parametrization you can use dbutils.widgets (https://docs...
- 2 kudos
- 5729 Views
- 2 replies
- 4 kudos
Why does Databricks SQL drop ending 0 in decimal data type
All,I have a column, RateAdj that is defined as DECIMAL(15,5) and I can see that the value is 4.00000, but when this gets inserted into my table it shows as just 4.%sql SELECT LTRIM(RTRIM(IFNULL(FORMAT_NUMBER(RateADJ, '0.00000'), '0.00000')))This i...
- 5729 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Greg Bradley​ Hope everything is going great.Checking in. If @Uma Maheswara Rao Desula​ answer helped, would you let us know and mark the answer as best? If not, would you be happy to give us more information?Thanks!
- 4 kudos
- 3137 Views
- 1 replies
- 0 kudos
Resolved! Error Using spark.catalog.dropTempView()
I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a not...
- 3137 Views
- 1 replies
- 0 kudos
- 0 kudos
spark.sql("DROP TABLE "+prefix_updates) spark.sql("DROP TABLE "+prefix_main)Fixed it for me.
- 0 kudos
| User | Count |
|---|---|
| 1615 | |
| 783 | |
| 349 | |
| 287 | |
| 277 |
