- 4604 Views
- 7 replies
- 1 kudos
Autoloader Schema Hint are not taken into consideration in schema file
I am using Autoloader with Schema Inference to automatically load some data into S3.I have one column that is a Map which is overwhelming Autoloader (it tries to infer it as struct -> creating a struct with all keys as properties), so I just use a sc...
- 4604 Views
- 7 replies
- 1 kudos
- 1 kudos
Sorry, I didn't mean that your solution is poorly designed. I was only referring to the one of the main definitions of your bronze layer: You want to have a defined and optimized data layout, which is source-driven at the same time. In other words: ...
- 1 kudos
- 7619 Views
- 3 replies
- 5 kudos
Resolved! Support for Varchar data type
In the official documentation for supported data types, Varchar is not listed. But in the product, we are allowed to create a field of varchar data type. We are building an integration with Databricks and we are confused if we should support operatio...
- 7619 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @phanindra ,To be precise, Delta Lake format is based on parquet files. For strings, Parquet only has one data type: StringTypeSo, basically varchar(n) data type under the hood is represented as string with check constraint on the length of the st...
- 5 kudos
- 893 Views
- 0 replies
- 1 kudos
Navigating the Future: Key Questions for Implementing Production-Quality GenAI
"The Big Book of Generative AI from Databricks"https://lnkd.in/dR4VuEyQprovides a comprehensive guide to understanding and implementing Generative AI (GenAI) effectively within an enterprise. Here are some critical questions that businesses should co...
- 893 Views
- 0 replies
- 1 kudos
- 18014 Views
- 13 replies
- 10 kudos
Spark tasks too slow and not doing parellel processing
Hi,I have spark job which is processing large data set, its taking too long to process the data. In Spark UI, I can see its running 1 tasks out of 9 tasks. Not sure how to run this in parellel. I have already mentioned auto scaling and providing upto...
- 18014 Views
- 13 replies
- 10 kudos
- 10 kudos
Will it be any different if using Spark but within Azure, i.e. faster?
- 10 kudos
- 2533 Views
- 3 replies
- 2 kudos
showing only a limited number of lines from the CSV file
Expected no of lines is - 16400 Showing only 20 No of records Script spark.conf.set( "REDACTED", "REDACTED" ) # File location file_location = "REDACTED" # Read in the data to dataframe df df = spark.read.format("CSV").option("inferSchema",...
- 2533 Views
- 3 replies
- 2 kudos
- 2 kudos
hi, pls look help mespark.conf.set( "REDACTED", "REDACTED")# File locationfile_location = "REDACTED"# Read in the data to dataframe dfdf = spark.read.format("CSV").option("inferSchema", "true").option("header", "true").option("delimiter", ",")...
- 2 kudos
- 658 Views
- 0 replies
- 0 kudos
Error 301 Moved Permanently in cells of plotting
Hi, I created a workbook for academic purposes and had completed it... from one moment to the next all the plot cells of charts (and only those) started returning the following error and I really don't know how to solve it by today.Failed to store th...
- 658 Views
- 0 replies
- 0 kudos
- 886 Views
- 1 replies
- 0 kudos
Replace one tag in a Jason file in the data bricks table .
There is a column (src_json) in emp_table . I need to replace (ages to age )in each json in the src_json column in emp_table.Can you pls suggest the best way to do it .
- 886 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Bhabs ,You can do it in following way (assuming that src_json contains json string):from pyspark.sql import SparkSession from pyspark.sql.functions import col, expr spark = SparkSession.builder.appName("Replace JSON Keys").getOrCreate() data = ...
- 0 kudos
- 2104 Views
- 1 replies
- 0 kudos
AutoLoader File Notification Setup on AWS
I’m encountering issues setting up Databricks AutoLoader in File Notification mode. The error seems to be related to UC access to the S3 bucket. I have tried running it on a single-node dedicated cluster but no luck.Any guidance or assistance on reso...
- 2104 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks @Retired_mod. I have reviewed all the steps mentioned, including the IAM policy, as per the setup guide. I believe the permissions granted are sufficient for the setup.To validate the permissions, I used IAM credentials with Admin privileges i...
- 0 kudos
- 1318 Views
- 0 replies
- 0 kudos
Error: the Service Account Key in storage credential is not configured correctly
We have databricks on GCP. Streamings are running 24/7, storage credentials and external location are created as we are using managed unity catalog. We get random error, somewhere are around mid night (UTC). Here is trace of Error,ERROR MicroBatchExe...
- 1318 Views
- 0 replies
- 0 kudos
- 2411 Views
- 2 replies
- 2 kudos
Job aborted stage failure java.sql.SQLRecoverableException: IO Error: Connection reset by peer
While ingesting data from Oracle to databricks(writing into ADLS) using jdbc I am getting connection reset by peer error when ingesting a large table which has millions of rows.I am using oracle sql developer and azure databricks.I tried every way li...
- 2411 Views
- 2 replies
- 2 kudos
- 2 kudos
Try using this code .import pyspark from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder.appName("OracleToDatabricks").getOrCreate() # Oracle connection properties conn = "jdbc:oracle:thin:@//<host>:<port>/<s...
- 2 kudos
- 7703 Views
- 2 replies
- 2 kudos
Resolved! Upload file from local file system to DBFS (2024)
Recent changes to the worskpace UI (and introduction of Unity Catalog) seem to have discretely sunset the ability to upload data directly to DBFS from the local Filesystem using the UI (NOT the CLI)I want to be able to load a raw file (no matter the ...
- 7703 Views
- 2 replies
- 2 kudos
- 3222 Views
- 2 replies
- 0 kudos
Unity Catalog 2.1 API Not Returning SQL Function/View Dependencies
Hi all,I have a problem with reading responses generated by Unity Catalog API 2.1 as they are missing fields that are otherwise described in the specification:List functions - The fields routine_dependencies, return_params, and input_params are missi...
- 3222 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod , I have the same issue as @DatabricksHero .View dependencies are not returned. Could you clarify the usage of this API call?
- 0 kudos
- 2134 Views
- 2 replies
- 4 kudos
date type conversion error
Hello,I am trying to convert the date in MM/dd/yyyy format. So I am first using the date_format function and converting the date into MM/dd/yyyy. So it becomes string. However, my use case is to have the data as date. so I am again converting the str...
- 2134 Views
- 2 replies
- 4 kudos
- 4 kudos
Check with this method if it works.# Convert date to MM/dd/yyyy format (string) df = df.withColumn("formatted_date", date_format("date", "MM/dd/yyyy")) # Convert string back to date df = df.withColumn("converted_date", to_date("formatted_date", "MM/...
- 4 kudos
- 4028 Views
- 4 replies
- 2 kudos
Error while trying to install jdbc8.jar
Hi,I am attempting to connect to an Oracle server. I tried to install the ojdbc8.jar library, but I encountered an error: "Library installation attempted on the driver node of cluster 0718-101257-h5k9c5ud failed. Please refer to the following error m...
- 4028 Views
- 4 replies
- 2 kudos
- 7612 Views
- 7 replies
- 1 kudos
Resolved! Datbricks JDK 17 upgrade error
We tried upgrading to JDK 17Using Spark version 3.0.5 and runtime 14.3 LTSGetting this exception using parallelstream()With Java 17 I am not able to parallel process different partitions at the same time. This means when there is more than 1 partiti...
- 7612 Views
- 7 replies
- 1 kudos
- 1 kudos
Anyways - thanks for your response - We found a workaround for this error and JDK 17 is actually working - it appears faster than JDK 8
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
