- 11319 Views
- 2 replies
- 1 kudos
Whats the difference between magic commands %pip and %sh pip
In Databricks you can do either %pipor %sh pipWhats the difference? Is there a recommended approach?
- 11319 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey there, User16776431030.Great question about those magic commands in Databricks! Let me shed some light on this mystical matter.The %pip and %sh pip commands may seem similar on the surface, but they're quite distinct in their powers. %sh pip is l...
- 1 kudos
- 5822 Views
- 4 replies
- 1 kudos
- 5822 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, it is possible to connect databricks to a kerberized hbase cluster. The attached article explains the steps. It consists of setting up a kerberos client using a keytab in the cluster nodes, installing the hbase-spark integration library, and set...
- 1 kudos
- 15436 Views
- 5 replies
- 6 kudos
Snowflake connection to Databricks error
When I am trying to read snowflake table from my databricks notebook, it is giving the error as:df1.read.format("snowflake") \.options(**options) \.option("query", "select * from abc") \.save()Getting below errorjava.sql.SQLException: No suitable dri...
- 15436 Views
- 5 replies
- 6 kudos
- 6683 Views
- 2 replies
- 1 kudos
Resolved! If I know the workspaceID, how can I figure out the the respective Databricks workspace URL?
- 6683 Views
- 2 replies
- 1 kudos
- 1 kudos
Is this random number not possible to extract from the notebook context? It is available in the browser_hash but that is not populated when running a job.Is this random number static or does it change over time? If it is static, it can then be hardco...
- 1 kudos
- 13850 Views
- 4 replies
- 1 kudos
Resolved! How to connect to DB2-AS400?
What are the steps needed to connect to a DB2-AS400 source to pull data to lake using Databricks? I believe it requires establishing a jdbc connection, but I couldnot find much details online
- 13850 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Ajay Menon​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 1 kudos
- 12353 Views
- 2 replies
- 5 kudos
Cannot create job through Jobs API
import requests import json instance_id = 'abcd.azuredatabricks.net' api_version = '/api/2.0' api_command = '/jobs/create' url = f"https://{instance_id}{api_version}{api_command}" headers = {'Authorization': 'Bearer myToken'} params = { "settings...
- 12353 Views
- 2 replies
- 5 kudos
- 5 kudos
@keenan_jones7​ I had the same problem today. It looks like you've copied and pasted the JSON that Databricks displays in the GUI when you select View JSON from the dropdown menu when viewing a job.In order to use that JSON in a request to the Jobs ...
- 5 kudos
- 3509 Views
- 3 replies
- 4 kudos
Resolved! Data size inflates massively while ingesting
GoalImport and consolidate GBs / TBs of local data in 20-mb chunk parquet files into Databricks / Delta lake / partitioned tables.What I've DoneI took a small subset of data, roughly 72.5 GB and ingested using streaming below. The data is already seq...
- 3509 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Erik Louie​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 7254 Views
- 3 replies
- 6 kudos
Question on Databricks Spark online proctored exam
Some silly questions folks. I took online proctored Databricks spark certification couple of days back and my unofficial result was pass. I received a mail that it might https://speedtest.vet/ take upto one week to receive the certification, if awar...
- 7254 Views
- 3 replies
- 6 kudos
- 6 kudos
better would have been to ask for permission before drinking. I can share my exp. My mobile alarm started buzzing during the exam, I requested the moderator, he then paused the exam and asked me to take my laptop to the mobile and then to switch off,...
- 6 kudos
- 3343 Views
- 1 replies
- 5 kudos
org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets
- 3343 Views
- 1 replies
- 5 kudos
- 8897 Views
- 5 replies
- 5 kudos
Connect a BI Tool: How do I access my lakehouse data from my BI tool?
You can find a rich ecosystem of tools that allow you to work with all your data in-place and deliver real-time business insights faster.This post will help you connect your existing tools like dbt, Fivetran, PowerBI, Tableau or SAP to ingest, transf...
- 8897 Views
- 5 replies
- 5 kudos
- 5 kudos
Hello Taha, here is a fairly recent video provided by Databricks on conncecting Power BI : Demo Video: Connect to Power BI Desktop from Databricks - YouTube
- 5 kudos
- 17981 Views
- 3 replies
- 4 kudos
java.lang.NoClassDefFoundError: scala/Product$class
Hi I am getting "java.lang.NoClassDefFoundError: scala/Product$class" error while using Deequ 1.0.5 version. Please suggest fix to this problem or any work around Error Py4JJavaError Traceback (most recent call last) <command-2625366351750561> in...
- 17981 Views
- 3 replies
- 4 kudos
- 17759 Views
- 2 replies
- 3 kudos
Resolved! Default Query Limit 1000
By default, we return back up to 1000 query results when a user runs a cell in Databricks. E.g., if you run display(storeData) and you have ten million customers, the UI will show the first 1000 results. If you graph that by age of customer, similarl...
- 17759 Views
- 2 replies
- 3 kudos
- 3 kudos
This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down.
- 3 kudos
- 3628 Views
- 2 replies
- 2 kudos
How to configure Databricks token inside Docker File?
I have a docker file where I want toDownload the Databricks CLIConfigure the CLI by adding a host and tokenAnd then running a python file that hits the Databricks tokenI am able to install the CLI in the docker image, and I have a working python file...
- 3628 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi I was facing same issue and searching for the solution but didnt get it, and now after working on it i have the solution if you want to access databricks models/download_artifacts using hostname and access token like how you do on databricks cli ...
- 2 kudos
- 9005 Views
- 1 replies
- 6 kudos
How can I simplify my data ingestion by processing the data as it arrives in cloud storage?
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables. Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delt...
- 9005 Views
- 1 replies
- 6 kudos
- 6 kudos
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables.Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delta...
- 6 kudos
- 2461 Views
- 2 replies
- 2 kudos
What is the long-term plan for the official Databricks Docker Containers?
Hi, I'd like to ask you, how much resources do you plan to dedicate to the maintenance/development of the official Databricks Docker images, please? Do you have a view on the longer-term plan for these docker images? It seems to be maintained, but i...
- 2461 Views
- 2 replies
- 2 kudos