"The Big Book of Generative AI from Databricks"https://lnkd.in/dR4VuEyQprovides a comprehensive guide to understanding and implementing Generative AI (GenAI) effectively within an enterprise. Here are some critical questions that businesses should co...
Hi,I have spark job which is processing large data set, its taking too long to process the data. In Spark UI, I can see its running 1 tasks out of 9 tasks. Not sure how to run this in parellel. I have already mentioned auto scaling and providing upto...
Will it be any different if using Spark but within Azure, i.e. faster?
Expected no of lines is - 16400 Showing only 20 No of records Script spark.conf.set( "REDACTED", "REDACTED" ) # File location file_location = "REDACTED" # Read in the data to dataframe df df = spark.read.format("CSV").option("inferSchema",...
hi, pls look help mespark.conf.set( "REDACTED", "REDACTED")# File locationfile_location = "REDACTED"# Read in the data to dataframe dfdf = spark.read.format("CSV").option("inferSchema", "true").option("header", "true").option("delimiter", ",")...
Hi, I created a workbook for academic purposes and had completed it... from one moment to the next all the plot cells of charts (and only those) started returning the following error and I really don't know how to solve it by today.Failed to store th...
There is a column (src_json) in emp_table . I need to replace (ages to age )in each json in the src_json column in emp_table.Can you pls suggest the best way to do it .
Hi @Bhabs ,You can do it in following way (assuming that src_json contains json string):from pyspark.sql import SparkSession from pyspark.sql.functions import col, expr spark = SparkSession.builder.appName("Replace JSON Keys").getOrCreate() data = ...
I’m encountering issues setting up Databricks AutoLoader in File Notification mode. The error seems to be related to UC access to the S3 bucket. I have tried running it on a single-node dedicated cluster but no luck.Any guidance or assistance on reso...
Thanks @Retired_mod. I have reviewed all the steps mentioned, including the IAM policy, as per the setup guide. I believe the permissions granted are sufficient for the setup.To validate the permissions, I used IAM credentials with Admin privileges i...
We have databricks on GCP. Streamings are running 24/7, storage credentials and external location are created as we are using managed unity catalog. We get random error, somewhere are around mid night (UTC). Here is trace of Error,ERROR MicroBatchExe...
While ingesting data from Oracle to databricks(writing into ADLS) using jdbc I am getting connection reset by peer error when ingesting a large table which has millions of rows.I am using oracle sql developer and azure databricks.I tried every way li...
Try using this code .import pyspark from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder.appName("OracleToDatabricks").getOrCreate() # Oracle connection properties conn = "jdbc:oracle:thin:@//<host>:<port>/<s...
Recent changes to the worskpace UI (and introduction of Unity Catalog) seem to have discretely sunset the ability to upload data directly to DBFS from the local Filesystem using the UI (NOT the CLI)I want to be able to load a raw file (no matter the ...
Hi all,I have a problem with reading responses generated by Unity Catalog API 2.1 as they are missing fields that are otherwise described in the specification:List functions - The fields routine_dependencies, return_params, and input_params are missi...
Hi @Retired_mod , I have the same issue as @DatabricksHero .View dependencies are not returned. Could you clarify the usage of this API call?
Hello,I am trying to convert the date in MM/dd/yyyy format. So I am first using the date_format function and converting the date into MM/dd/yyyy. So it becomes string. However, my use case is to have the data as date. so I am again converting the str...
Check with this method if it works.# Convert date to MM/dd/yyyy format (string) df = df.withColumn("formatted_date", date_format("date", "MM/dd/yyyy")) # Convert string back to date df = df.withColumn("converted_date", to_date("formatted_date", "MM/...
Hi,I am attempting to connect to an Oracle server. I tried to install the ojdbc8.jar library, but I encountered an error: "Library installation attempted on the driver node of cluster 0718-101257-h5k9c5ud failed. Please refer to the following error m...
We tried upgrading to JDK 17Using Spark version 3.0.5 and runtime 14.3 LTSGetting this exception using parallelstream()With Java 17 I am not able to parallel process different partitions at the same time. This means when there is more than 1 partiti...
Anyways - thanks for your response - We found a workaround for this error and JDK 17 is actually working - it appears faster than JDK 8
I have a notebook that, as a first step, needs to download and install some drivers. The actual code is this:%sh# Install gdebi command line toolapt-get -y install gdebi-core# Install Posit professional driverscurl -LO https://cdn.rstudio.com/drivers...
Hi @mb1234 ,What error did you get?Edit: I've checked and it worked in job
Hi!As suggested by Databricks, we are working with Databricks from VSCode using Databricks bundles for our deployment and using the VSCode Databricks Extension and Databricks Connect during development.However, there are some limitations that we are ...
Thank you for the detailed reply, @Retired_mod and the great question @pernilak!I would also like to code and debug in VS Code while all the code in my Jupyter notebooks can be executed on a databricks cluster cell by cell with access to the data in ...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 557 | |
| 349 | |
| 287 |
