- 4536 Views
- 2 replies
- 2 kudos
Resolved! DLT Unity catalog schema no storage location mention details
Hi TeamAs part of an earlier That I did with Databricks team , I got the info for that if one wants to dump the data in unity catalog schema from a DLT pipeline , The specific schema's Storage location must not be specified . else he DLT pipeline wil...
- 4536 Views
- 2 replies
- 2 kudos
- 2 kudos
Thanks @Walter_C for the explanation and confirming the understanding. Really Appreaciate
- 2 kudos
- 8984 Views
- 0 replies
- 1 kudos
5 tips to get the most out of your Databricks Assistant
Back in July, we released the public preview of the new Databricks Assistant, a context-aware AI assistant available in Databricks Notebooks, SQL editor and the file editor that makes you more productive within Databricks, including: Generate SQL or ...
- 8984 Views
- 0 replies
- 1 kudos
- 2517 Views
- 0 replies
- 0 kudos
Bringing breakthrough data intelligence to industries
Gen AI for All: Empowering Every Role Across Industries The new frontier of data intelligence is here. As more companies pursue industry-changing transformations, they face the same monumental challenge: how to democratize data and AI. In this new r...
- 2517 Views
- 0 replies
- 0 kudos
- 2632 Views
- 1 replies
- 0 kudos
Spark doesn't register executors when new workers are allocated
Our pipelines sometimes get stuck (example).Some workers get decommissioned due to spot termination and then the new workers get added. However, after (1) Spark doesn't notice new executors: And I don't know why. I don't understand how to debug this,...
- 2632 Views
- 1 replies
- 0 kudos
- 0 kudos
@ivanychev - Firstly, New workers are added and spark notice them hence, there is an init script logging in the event log stating the init script ran on the newly added workers. For debugging, please check the Spark UI - executor tab. Secondly, Fo...
- 0 kudos
- 2264 Views
- 1 replies
- 1 kudos
Ifichangethetimestampformat from yyyy-MM-dd hh-mm-ss to MM-dd-yyyy hh-mm-ssinPostgrestableisthatfine
- 2264 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, You can check the function https://docs.databricks.com/en/sql/language-manual/functions/date_format.html, let us know if this helps.
- 1 kudos
- 5873 Views
- 1 replies
- 0 kudos
How to query sql warehouse tables with spark?
Hey there... I managed to query my data following this guide https://learn.microsoft.com/en-us/azure/databricks/dev-tools/python-sql-connectorusing databricks sql#!/usr/bin/env python3from databricks import sqlwith sql.connect(server_hostname = "adb-...
- 5873 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mobe - Please refer to the github link for more examples - https://github.com/databricks/databricks-sql-python/blob/main/examples. Thanks,Shan
- 0 kudos
- 3977 Views
- 2 replies
- 1 kudos
Getting 'No GCP Marketplace token provided' error while signing up from GCP marketplace.
Hey guys,I was trying to sign up to the 14 day free trial from GCP marketplace. When I click 'SIGN UP WITH DATABRICKS', I get the error below.HTTP ERROR 401Problem accessing /sign-up. Reason: No GCP Marketplace token provided. Please start over fr...
- 3977 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks Walter,I have the IAM permissions in place and also have a valid billing account.However, I keep getting the same error regarding the missing Marketplace token. I am clicking the 'SIGN UP WITH DATABRICKS' button from the GCP UI, so am not sure...
- 1 kudos
- 1913 Views
- 2 replies
- 0 kudos
HELP opening notebook displays blank, creating new one gives and error and other issues
Hi,SituationI just literally started using data bricks. I created a workspace, a cluster and uploaded a notebook. But my workspace doesn’t seem to function correctly at the moment.I will attach what it looks like when I try to open a notebookopening ...
- 1913 Views
- 2 replies
- 0 kudos
- 0 kudos
UPDATEI have downloaded chrome and this does not happen for it as well
- 0 kudos
- 1351 Views
- 1 replies
- 0 kudos
Databricks Widget
Hi,I was previously working databricks runtime 10.0 and now just upgraded to 13.0 runtime.I was using dashboard to display the widgets. Before it was just showing the widget label, but now it shows the widget name below it as well. Also it shows the ...
- 1351 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @aman_yadav007, which widget type did you use? Can you please try a different widget type or check the widget type and its arguments from this example: https://docs.databricks.com/en/notebooks/widgets.html#databricks-widgets
- 0 kudos
- 3264 Views
- 2 replies
- 0 kudos
DLT pipeline unity catalog error
Hi Everyone, I'm getting this error while running DLT pipeline in UCFailed to execute python command for notebook 'sample/delta_live_table_rules.py' with id RunnableCommandId(5596174851701821390) and error AnsiResult(,None, Map(), Map(),List(),List(...
- 3264 Views
- 2 replies
- 0 kudos
- 0 kudos
I get a similar error, when there is a mistake in the @dlt.table() definition for a table. In my case the culprit is usually the path.
- 0 kudos
- 2295 Views
- 2 replies
- 0 kudos
Unable to migrate an empty parquet table to delta lake in Databricks
I'm trying to convert my Databricks Tables from Parquet to Delta. While most of the tables have data and are successfully converted to delta some of the empty parquet tables fail with an error message as below -CONVERT TO DELTA <schema-name>.parquet_...
- 2295 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello Bharathi, Ideally the ETL job should not generate the empty parquet files in the respective location as it's an overhead to read the empty file and it's a not best practice.Assuming this can be easily fix in ETL job while getting the rows count...
- 0 kudos
- 786 Views
- 0 replies
- 0 kudos
L'importance de Databricks dans le SEO
Le SEO est un domaine dynamique et complexe qui évolue constamment avec les technologies et les algorithmes de recherche. L'utilisation de Databricks, une plateforme d'analyse basée sur le cloud, a révolutionné la manière dont les spécialistes du SEO...
- 786 Views
- 0 replies
- 0 kudos
- 7987 Views
- 2 replies
- 0 kudos
Read CSV files in Azure Databricks notebook, how to read data when columns in CSV files are in the w
I have a task to revise CSV ingestion in Azure Databricks. The current implementation uses the below settings: source_query = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "csv") .schema(defined_schema) .option(...
- 7987 Views
- 2 replies
- 0 kudos
- 0 kudos
Also, I am looking for a solution that works with both correct files and malformed files using PySpark.
- 0 kudos
- 10412 Views
- 4 replies
- 0 kudos
Resolved! DatabaseError: (databricks.sql.exc.ServerOperationError) [UNBOUND_SQL_PARAMETER]
Hi,I am trying to connect my database through LLM and expecting to receive a description of the table and 1st 3 rows from the table. from langchain.agents import create_sql_agent from langchain.agents.agent_toolkits import SQLDatabaseToolkit from la...
- 10412 Views
- 4 replies
- 0 kudos
- 0 kudos
This is not databricks issue but from langchain. A PR has been raised to solve this: One workaround that worked is: https://github.com/langchain-ai/langchain/issues/11068 setting sample_rows_in_table_info to 0 when calling SQLDatabase.from_databricks...
- 0 kudos
- 11687 Views
- 2 replies
- 1 kudos
Creating High Quality RAG Applications with Databricks
Retrieval-Augmented-Generation (RAG) has quickly emerged as a powerful way to incorporate proprietary, real-time data into Large Language Model (LLM) applications. Today we are excited to launch a suite of RAG tools to help Databricks users build hig...
- 11687 Views
- 2 replies
- 1 kudos
- 1 kudos
It seems like you're sharing an announcement or promotional content related to Databricks and their launch of a suite of tools for Retrieval-Augmented-Generation (RAG) applications. These tools are aimed at helping Databricks users build high-quality...
- 1 kudos
