- 5884 Views
- 4 replies
- 0 kudos
Can we assigee default value in select columns in Spark sql when the column is not present?
Im reading avro file and loading into table. The avro data is nested data.Now from this table im trying to extract the necessary elements using spark sql. Using explode function when there is array data. Now the challenge is there are cases like the ...
- 5884 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @manoj kumar​ An easiest way would be to make use of unmanaged delta tables and while loading data into the path of that table, you can enable mergeSchema to be true. This handles all the schema differences, incase column is not present as null an...
- 0 kudos
- 6357 Views
- 5 replies
- 3 kudos
Resolved! Scope creation in Databricks or Confluent?
Hello I am a newbie in this field and trying to access confluent kafka stream in Databricks Azure based on a beginner's video by Databricks. I have a free trial of Databricks cluster right now. When I run the below notebook, it errors out on line 5 o...
- 6357 Views
- 5 replies
- 3 kudos
- 3 kudos
For testing, create without secret scope. It will be unsafe, but you can post secrets as strings in the notebook for testing. Here is the code which I used for loading data from confluent:inputDF = (spark .readStream .format("kafka") .option("kafka.b...
- 3 kudos
- 6032 Views
- 3 replies
- 0 kudos
Databricks workspace (with managed VNET) upgrade to premium failing
I am trying to upgrade our Databricks workspace from standard to premium but running into issues. The workspace is currently deployed in a managed VNET.I tried the migration tool as well as just re-creating a premium workspace with the same parameter...
- 6032 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, I have same situation when trying to upgrade from Standard to Premium on Azure.My error: "ConflictWithNetworkIntentPolicy","message":"Found conflicts with NetworkIntentPolicy. Details: Subnet or Virtual Network cannot have resources or properties...
- 0 kudos
- 1715 Views
- 2 replies
- 1 kudos
Lineage - It would be nice if the lineage in Unity would allow for API calls that could add additional lineage information, somehow. I am not certain...
Lineage - It would be nice if the lineage in Unity would allow for API calls that could add additional lineage information, somehow. I am not certain exactly what would be nice. But some sort of feature to include source systems in it.
- 1715 Views
- 2 replies
- 1 kudos
- 1 kudos
Pureview is quite popular to be integrated to solve this issue. I think lineage in the unity catalog is designed to be auto-generated. I know there are information tables, but I never manually manipulated them.
- 1 kudos
- 5309 Views
- 3 replies
- 1 kudos
linking filters from different Databricks SQL queries in a Dashboard
I am having to use Databricks SQL dashboard for some analysis, it seems very clunky. If i have multiple queries, is it possible to apply the same filters to all the queries in the dashboard or do i have to duplicate the filters for each query in the ...
- 5309 Views
- 3 replies
- 1 kudos
- 1 kudos
Same issue here. According the docs, using query filters with the same name and values should result in a single dashboard filter. However, filters are duplicated. I also tried using this setting but no success
- 1 kudos
- 2617 Views
- 3 replies
- 1 kudos
Resolved! Do we have any certificate voucher for the data bricks session in the upcoming days
Hi Team,Do we have any program for certificate vouchers for the data bricks session in upcoming days
- 2617 Views
- 3 replies
- 1 kudos
- 1 kudos
@Vidula Khanna​ I got this link for the certificate voucher register link.https://docs.google.com/presentation/d/1sy5hSSnFtncrpYY1EYi0WMsDkJK0dYk9iKBAeeAha8E/edit#slide=id.g1ade45a9cd6_0_543
- 1 kudos
- 11368 Views
- 9 replies
- 7 kudos
Calling a python function (def) in databricks
Not sure if I'm missing something here, but running a task outside of a python function runs much much quicker than executing the same task inside a function. Is there something I'm missing with how spark handles functions? 1) def task(x): y = dostuf...
- 11368 Views
- 9 replies
- 7 kudos
- 7 kudos
don't use python normal function use UDF in pyspark so that will be faster
- 7 kudos
- 8664 Views
- 3 replies
- 3 kudos
Resolved! Is timestamp difference always INTERVAL DAY TO SECOND?
My observations show that timestamp difference has type of INTERVAL DAY TO SECONDS:select typeof(getdate() - current_date()) ----------------------------------------- interval day to secondBut is it guaranteed? Can it be DAY TO MINUTE or, say, YEAR T...
- 8664 Views
- 3 replies
- 3 kudos
- 3 kudos
you can check here for given example: https://docs.databricks.com/sql/language-manual/functions/minussign.htmlthis might help to you.
- 3 kudos
- 7151 Views
- 4 replies
- 3 kudos
Generate sh2 hashkey while loading files to delta table
I have files in azure data lake. I am using autoloader to read the incremental filesfiles don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.In this case i want to ...
- 7151 Views
- 4 replies
- 3 kudos
- 4984 Views
- 3 replies
- 3 kudos
- 4984 Views
- 3 replies
- 3 kudos
- 3 kudos
Df.write.format('jdbc').options( url='jdbc:oracle:thin:@192.168.11.100:1521:ORCL', driver='oracle.jdbc.driver.OracleDriver', dbtable='testschema.test', user='testschema', password='password').mode('overwrite').save()try ...
- 3 kudos
- 15359 Views
- 5 replies
- 6 kudos
split character string in cell with sql
I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...
- 15359 Views
- 5 replies
- 6 kudos
- 6 kudos
you need to use explode functionhttps://stackoverflow.com/questions/61070630/spark-explode-column-with-json-array-to-rows
- 6 kudos
- 6676 Views
- 2 replies
- 3 kudos
How to read enum[] (enum of array) datatype from postgres using spark
We are trying to read a column which is enum of array datatype from postgres as string datatype to target. We could able to achieve this by expilcitly using concat function while extracting like belowval jdbcDF3 = spark.read .format("jdbc") .option(...
- 6676 Views
- 2 replies
- 3 kudos
- 3 kudos
You can try custom schema for JDBC read.option("customSchema", "colname STRING")
- 3 kudos
- 2905 Views
- 1 replies
- 1 kudos
Resolved! Apache Spark Programming course -- repo notebooks don't seem to match up with content
I have no idea what I'm doing wrong. I synched this repo <https://github.com/databricks-academy/apache-spark-programming-with-databricks> as instructed. You can see that Notebook 4.1 is query optimization in the 'published' branch.https://github.com/...
- 2905 Views
- 1 replies
- 1 kudos
- 1 kudos
It happened to me also in the past as courses are constantly updated. You can inform the training department through https://help.databricks.com/s/contact-us?ReqType=training
- 1 kudos
- 3317 Views
- 2 replies
- 4 kudos
Resolved! How to Kafka configured on your PC with Databricks?
I'm working on the case to configure Kafka that is installed on my machine (Laptop) & I want to connect it with my Databricks account hosted on the AWS cloud.Secondly, I have CSV files that I want to use for real-time processing from Kafka to Databri...
- 3317 Views
- 2 replies
- 4 kudos
- 4 kudos
For CSV, you need just to readStream in the notebook and append output to CSV using forEachBatch method.Your Kafka on PC needs to have the public address or you need to set AWS VPN and connect from your laptop to be in the same VPC as databricks.
- 4 kudos
- 6667 Views
- 2 replies
- 0 kudos
Databricks SQL Visualizations - export
How can I export Databricks SQL Dashboards as a .pdf?
- 6667 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello, I'm trying to export my dashboard to PDF but when I run the export, none of the data is exported, only blank visuals. I would appreciate any help in getting this resolved.Dashboard in DBX: Exported PDF:
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 529 | |
| 349 | |
| 287 |
