- 7686 Views
- 9 replies
- 7 kudos
Calling a python function (def) in databricks
Not sure if I'm missing something here, but running a task outside of a python function runs much much quicker than executing the same task inside a function. Is there something I'm missing with how spark handles functions? 1) def task(x): y = dostuf...
- 7686 Views
- 9 replies
- 7 kudos
- 7 kudos
don't use python normal function use UDF in pyspark so that will be faster
- 7 kudos
- 5284 Views
- 3 replies
- 3 kudos
Resolved! Is timestamp difference always INTERVAL DAY TO SECOND?
My observations show that timestamp difference has type of INTERVAL DAY TO SECONDS:select typeof(getdate() - current_date()) ----------------------------------------- interval day to secondBut is it guaranteed? Can it be DAY TO MINUTE or, say, YEAR T...
- 5284 Views
- 3 replies
- 3 kudos
- 3 kudos
you can check here for given example: https://docs.databricks.com/sql/language-manual/functions/minussign.htmlthis might help to you.
- 3 kudos
- 4741 Views
- 4 replies
- 3 kudos
Generate sh2 hashkey while loading files to delta table
I have files in azure data lake. I am using autoloader to read the incremental filesfiles don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.In this case i want to ...
- 4741 Views
- 4 replies
- 3 kudos
- 3321 Views
- 3 replies
- 3 kudos
- 3321 Views
- 3 replies
- 3 kudos
- 3 kudos
Df.write.format('jdbc').options( url='jdbc:oracle:thin:@192.168.11.100:1521:ORCL', driver='oracle.jdbc.driver.OracleDriver', dbtable='testschema.test', user='testschema', password='password').mode('overwrite').save()try ...
- 3 kudos
- 11730 Views
- 5 replies
- 6 kudos
split character string in cell with sql
I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...
- 11730 Views
- 5 replies
- 6 kudos
- 6 kudos
you need to use explode functionhttps://stackoverflow.com/questions/61070630/spark-explode-column-with-json-array-to-rows
- 6 kudos
- 3120 Views
- 2 replies
- 3 kudos
How to read enum[] (enum of array) datatype from postgres using spark
We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like belowval jdbcDF3 = spark.read .format("jdbc") .option(...
- 3120 Views
- 2 replies
- 3 kudos
- 3 kudos
You can try custom schema for JDBC read.option("customSchema", "colname STRING")
- 3 kudos
- 1894 Views
- 1 replies
- 1 kudos
Resolved! Apache Spark Programming course -- repo notebooks don't seem to match up with content
I have no idea what I'm doing wrong. I synched this repo <https://github.com/databricks-academy/apache-spark-programming-with-databricks> as instructed. You can see that Notebook 4.1 is query optimization in the 'published' branch.https://github.com/...
- 1894 Views
- 1 replies
- 1 kudos
- 1 kudos
It happened to me also in the past as courses are constantly updated. You can inform the training department through https://help.databricks.com/s/contact-us?ReqType=training
- 1 kudos
- 2273 Views
- 2 replies
- 4 kudos
Resolved! How to Kafka configured on your PC with Databricks?
I'm working on the case to configure Kafka that is installed on my machine (Laptop) & I want to connect it with my Databricks account hosted on the AWS cloud.Secondly, I have CSV files that I want to use for real-time processing from Kafka to Databri...
- 2273 Views
- 2 replies
- 4 kudos
- 4 kudos
For CSV, you need just to readStream in the notebook and append output to CSV using forEachBatch method.Your Kafka on PC needs to have the public address or you need to set AWS VPN and connect from your laptop to be in the same VPC as databricks.
- 4 kudos
- 3707 Views
- 2 replies
- 0 kudos
Databricks SQL Visualizations - export
How can I export Databricks SQL Dashboards as a .pdf?
- 3707 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, I'm trying to export my dashboard to PDF but when I run the export, none of the data is exported, only blank visuals. I would appreciate any help in getting this resolved.Dashboard in DBX: Exported PDF:
- 0 kudos
- 3622 Views
- 3 replies
- 0 kudos
Reading from /tmp no longer working
Since yesterday, reading a file copied into the cluster is no longer working.What used to work:blob = gcs_bucket.get_blob("dev/data.ndjson") -> worksblob.download_to_filename("/tmp/data-copy.ndjson") -> worksdf = spark.read.json("/tmp/data-copy.ndjso...
- 3622 Views
- 3 replies
- 0 kudos
- 0 kudos
I encountered this same issue, and figured out a fix!For some reason, it seems like only %sh cells can access the /tmp directory. So I just did...%sh ch /tmp/<file> /dbfs/<desired-location> and then accessed it form there using Spark.
- 0 kudos
- 3532 Views
- 4 replies
- 1 kudos
Resolved! I haven't received badge for completion
Hi,Today I completed the test for Lakehouse fundamentals by scored 85%, still I haven't received the badge through my email francis@intellectyx.comKindly let me know please !-Francis
- 3532 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi, I have same problem but you must wait up to 24 hours and you will receive the badge
- 1 kudos
- 1520 Views
- 3 replies
- 2 kudos
I haven't received certificate or badge after clearing data engineer associate exam
Hello Databricks Team,I completed my data engineer associate exam & passed the exam. But still now I haven't received certificate or badge. Also created help ticket, Ticket Number: #[00232273]​ Mail id : satishshravan888@gmail.comThanks,Satish
- 1520 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Satish Shravan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 2 kudos
- 13345 Views
- 3 replies
- 6 kudos
Resolved! "User not authorized" error when trying to deploy access connector on Azure
Hi,I have been trying to deploy Access Connector resource on Azure using Azure Pipelines (YAML) and a Bicep template but I cannot find a solution to this error:ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resou...
- 13345 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi,I fixed this issue by adding the service principal to the list of service principals in the Account Console. My guess is that after the access connector is created an API call is made to the Databricks account and the service principal making that...
- 6 kudos
- 3228 Views
- 2 replies
- 2 kudos
implement autoloader to ingest data into delta lake, i have 100 different tables with full load , append merge senarios
i want to implement autoloader to ingest data into delta lake from 5 different source systems and i have 100 different tables in each database how do we dynamically address this by using autoloader , trigger once option - full load , append merge sen...
- 3228 Views
- 2 replies
- 2 kudos
- 2 kudos
You can create a generic notebook that will be parametrized with the table name/source system and then just simply trigger notebook with different parameters (for each table/source system).For parametrization you can use dbutils.widgets (https://docs...
- 2 kudos
- 2365 Views
- 1 replies
- 0 kudos
Resolved! Error Using spark.catalog.dropTempView()
I have a set of Spark Dataframes that I convert into Temp Views to run Spark SQL with. Then, I delete them after my logic/use is complete. The delete step throws an odd error that I am not sure how to fix. Looking for some tips on fixing it. As a not...
- 2365 Views
- 1 replies
- 0 kudos
- 0 kudos
spark.sql("DROP TABLE "+prefix_updates) spark.sql("DROP TABLE "+prefix_main)Fixed it for me.
- 0 kudos