- 199 Views
- 2 replies
- 0 kudos
Filling a STRUCT field with a PreparedStatement in JDBC
Hello, I'm trying to fill a STRUCT field with a PreparedStatement in Java by giving a JSON string in the PreparedStatement.But it complains Cannot resolve "infos" due to data type mismatch: cannot cast "STRING" to "STRUCT<AGE: BIGINT, NAME: STRING>"....
- 199 Views
- 2 replies
- 0 kudos
- 0 kudos
Could you provide a sample of JSON string along with a code you're using? Otherwise it will be hard for us to help you.
- 0 kudos
- 207 Views
- 2 replies
- 3 kudos
Resolved! Difference between libraries dlt and dp
In all Databricks documentation, the examples use import dlt to create streaming tables and views. But, when generating sample Python code in ETL pipeline, the import in the sample is:import pyspark import pipelines as dpWhich one is the correct libr...
- 207 Views
- 2 replies
- 3 kudos
- 3 kudos
@yit Functionally, they are equivalent concepts (declarative definitions for streaming tables, materialized views, expectations, CDC, etc.). The differences you’ll notice are mostly naming/ergonomics:Module name:Databricks docs & most existing notebo...
- 3 kudos
- 328 Views
- 8 replies
- 3 kudos
Flattening the json in databricks
I have chatbot data I read adls json file in databricks and i stored the output in dataframeIn that table two columns contains json data but the data type is string1.content2.metadata Now i have to flatten the.data but i am not getting how to do tha...
- 328 Views
- 8 replies
- 3 kudos
- 3 kudos
Hi @szymon_dybczak I gave the wrong content json valueHere is the updated one could you please tell me the code for this it would be helpful for me you gave the code already but i am getting confused so please tell me for this { "activities": [ { "va...
- 3 kudos
- 3563 Views
- 5 replies
- 0 kudos
Connect Oracle Fusion (ERP / HCM) to Databricks
Any suggestion to connect Oracle Fusion (ERP/HCM) to Databricks?I have explored a few options including the use of Oracle Integration Cloud but it requires a lot of customization.
- 3563 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey Ralph,Orbit datajump is one of the good options in the market. Try it out.
- 0 kudos
- 859 Views
- 4 replies
- 0 kudos
bug with using parameters in a sql task
I am trying to make a sql task that runs using a serverless sql warehouse that takes a variable and uses that in the sql file that it is running in a serverless warehouse, however I am getting errors because databricks keeps formatting it first with ...
- 859 Views
- 4 replies
- 0 kudos
- 0 kudos
I have been trying to use templates as well and got the following string:LOCATION 's3://s3-company-data-' dev '-' 1122334455 '-eu-central-1/path_to_churn/main/'
- 0 kudos
- 262 Views
- 4 replies
- 1 kudos
Resolved! ExcelWriter and local files
I have a couple things going on here.First, to explain what I'm doing, I'm passing an array of objects in to a function that contain a dataframe per item. I want to write those dataframes to an excel workbook - one dataframe per worksheet. That part ...
- 262 Views
- 4 replies
- 1 kudos
- 1 kudos
Hello @data-grassroots! Were you able to resolve this? If any of the suggestions shared above helped, or if you found another solution, it would be great if you could mark it as the accepted solution or share your approach with the community.
- 1 kudos
- 191 Views
- 5 replies
- 4 kudos
Resolved! Self Dependency TumblingWindowTrigger in adf
Hey !I would like to migrate one ADF batch ingestion which has a TumblingWindowTrigger on top of the pipeline which pretty much check each 15 min if a file is landing, normally the files land in daily basis so will process it accordingly once in a d...
- 191 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi @szymon_dybczak ,sounds reasonable, will propone this approach. Thanks
- 4 kudos
- 526 Views
- 12 replies
- 17 kudos
Accessing Spark UI in free edition
Hello, is it possible to access Spark UI in free edition, I want to check task and stages.Ultimately I am working on how to check data skewness.
- 526 Views
- 12 replies
- 17 kudos
- 17 kudos
@szymon_dybczak @BS_THE_ANALYST Is there a specific guide or a flow to be a better databricks data engineer. I am learning as the topic comes up.Finding it really difficult to maintain a flow and I lose track.
- 17 kudos
- 104 Views
- 2 replies
- 1 kudos
Unable to Verify Account – "User is not a member of this workspace" Community addition..
I am encountering an issue during the verification process while trying to access my Databricks account.I received the verification code via email; however, when I attempt to verify it, I receive the following error message:"User is not a member of t...
- 104 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @perenehal! The error indicates that you haven’t created an account in the Databricks Community Edition. Currently, creating new accounts for Community Edition is not possible. However, if you already had an existing account and are still seein...
- 1 kudos
- 3225 Views
- 2 replies
- 1 kudos
How agentic ai is different from Ai agents
How agentic ai different from ai agents
- 3225 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey there! Let me break this down simply.What's an AI Agent?An AI agent is like a smart worker who follows instructions for one specific job. Think of a customer service chatbot that answers "Where's my order?" perfectly. But ask something complex li...
- 1 kudos
- 74 Views
- 1 replies
- 1 kudos
Spark Streaming Loading 1kto 5k rows only delta table
Hi Team,I have 4-5 millions of files in s3 files around 1.5gb data only with 9 million records, when i try to use autoloader to read the data using read stream and writing to delta table the processing is taking too much time, it is loading from 1k t...
- 74 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @bunny1174 ,You have 4-5 millions of files in s3 and their size is 1.5gb - this clearly indicates small files problem. You need compact those files to bigger size. There's no way your pipeline will be performant if you have such many files and the...
- 1 kudos
- 170 Views
- 7 replies
- 4 kudos
configure AWS authentication for serverless Spark
I only have an AWS Access Key ID and Secret Access Key, and I want to use this information to access S3.However, the official documentation states that I need to set the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID environment variables, but I cannot ...
- 170 Views
- 7 replies
- 4 kudos
- 4 kudos
Hi @liu ,The proper way is to go to your cluster and in advanced section you can set them up. In that way they will be scoped at cluster level. It's recommended to store values itself in a secret scopes as environment variables:Use a secret in a Spa...
- 4 kudos
- 71 Views
- 1 replies
- 1 kudos
is there another way to authen to azure databricks using MSI on Java
Hi I am try to connect to azure databricks using MSI on Java but on a documenthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/azure-miit saidThe Databricks SDK for Java has not yet implemented Azure managed identities authentication...
- 71 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @touchyvivace Unfortunately not, the documentation is up to date. In the Java SDK, MSI has not been implemented yet.And here's an open issue on github:[FEATURE] Add support for Azure Managed Identity authentication (system and user-assigned) · Iss...
- 1 kudos
- 96 Views
- 1 replies
- 2 kudos
Dev/Pie/Prd and the same workspace
Hi all!I'm appealing to all you folk who are clever than I for some advice on Databricks dev ops.I was asked by my team leader to expand our singular environment to a devops style dev/pie/prd system, potentially using Dabs to promote code to higher e...
- 96 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @IONA ,I guess you can still use DABs to simulate different environments on single workspace. In targets define 3 different environments but with the same for all of them (something similar to the picture below).Then your intuition is good - it's ...
- 2 kudos
- 138 Views
- 1 replies
- 1 kudos
ServiceNow LakeFlow Connector - Using TABLE API only for tables and NOT views
The current Databricks ServiceNow Lakeflow connector relies on ServiceNow REST TABLE API to capture data. And for some reason, it is unable to list a user defined view as a data source to be configured, even though ServiceNow user defined views are a...
- 138 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @ckanzabedian, have you checked out the documentation yet for the ServiceNow connector?https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-limits The link above is about the limits. I can't see a mention about ...
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 450 | |
| 349 | |
| 287 |
